Archive for January, 2015

Before it was called “DevOps”, it was just “magic”

Monday, January 26th, 2015

One thing I had forgotten in my time away from Amazon was that their code management, packaging, and deployment systems are magic. Seriously impressive internally built tools to manage a large, but usually apolllodecoupled codebase across thousands of servers in data centers around the world. Magic indeed… sometime black magic, but magic nonetheless.

There are pretty strict limits on what I can say about internal systems, but on the deployment piece at least the cat is out of the bag. Werner Vogels, Amazon CTO talks about Apollo, the Amazon deployment system here.

By comparison, the other massively scalable deployment system I know about is Microsoft’s Autopilo, which you can read about here,.  Autopilot is good for what it is, but Apollo has the advantage in many areas including interface (usability), configurability, packaging, and versioning, which is not surprising given its 10 year head start.

And those systems are just deployment.  My ability to search code, have code dependencies just handled for me (at development, build, and debug times), and to build code in a consistent trackable way is simply impressive at Amazon, reflecting Amazon’s long history in the software services space.

Finally my ability to request new servers at Amazon (generally EC2 virtual servers) is as easy going to a website and filling out a form.  I remember in my early days at Microsoft having to select SKUs and meet with a vendor to place the order. Then a week or so later I got a phone call telling me my servers were on a pallet on the data center loading dock…. what did I want done with them :-)

Amazon has done DevOps long before that word existed.  Like Microsoft, any company making the move into software services learns quickly how important that is.

How to communicate: Tools of the trade

Monday, January 19th, 2015

One of the biggest issues I see on struggling software teams (although this is not limited to software) is problems with communication.  Modern software is complex, and therefore our software teams can be complex.  There are many groups within and external to the team that require information for successful delivery

  • Developers
  • Testers
  • Engineering managers
  • Product managers
  • Business stakeholders
  • Customers
  • Upper level management

Here are my quick thoughts on HOW we choose to communicate.

This list is in ascending order – bias to use the ones at the top over the ones at the bottom

Discussion (meetings, phone, IM)

Best for complex updates and impactful events. Maximizes understanding and reduces conflict via interactive communication.

Have a whiteboard (or virtual equivalent) to ensure understanding around complex technical or business logic

Issue management / ticketing systems

Best for long term item tracking, history, and status snapshot (NOT good for conveying major changes unless accompanied by discussion)

Wiki / OneNote / shared collaboration documents

Best for capturing discussion outcomes and project status. May not need if tracking system offers rich enough experience to accommodate these.


Best for "simple" updates.

Email can point to Issues/Reports or Wiki.

OK to start discussions which require documents or spec review (providing pointers to those) and then complete discussion outside of email

I am obviously biased towards agile. The following Agile Manifesto values are represented here

Individuals and interactions over processes and tools
Customer collaboration over contract negotiation

Note on meetings

Meetings when done wrong go to the BOTTOM of the list. Specifically having too many people present who do not have a clear contribution to the meeting and/or meetings without clear goals. Bias for short meetings with small groups — a quick ad-hoc 2-person face to face (or virtual equivalent) often suffices. Then follow up with one of the other three communication tools to verify all are aligned on what was agreed, and to provide a history.

TiP is misunderstood – perhaps DDQ is Better

Monday, January 12th, 2015

I spent a long time talking to folks about the merits of a conscientious Testing in Production (TiP) strategy.  But I knew TiP had a bad rap.  I even shared the story of how some would mischaracterize it as a common and costly technical malpractice

While evangelizing TiP, I and my Microsoft colleagues would happily post this picture wherever we could

imageYet I knew the original poster was not so enthused with TiP.   Comments on TiP were supposing this was not a conscientious and risk-mitigated strategy, but instead devs behaving badly:

Then blame all issues on QA -_-

That’s our motto here. Doesn’t work to well in practice, actually.

Now I have returned to Amazon after spending 6 years at Microsoft.  From the following it looks like I have some education to do.


On the other hand, who can argue with Data-Driven Quality (DDQ).  (Except maybe a HiPPO).  DDQ is also more expansive than TiP, leveraging all data streams whether from production, customer research, or pre-release engineering.  So TiP was fun, but DDQ is the future.

Who is the HiPPO?

Thursday, January 8th, 2015


HiPPO stands for Highest Paid Person’s Opinion

HiPPO driven decision making is the opposite of data-driven decision making.  The “highest paid” person may be your boss, or the VP with his eye on the project, or even the CEO.  But no matter how many of those big bucks they are pulling down, it turns out that 2/3 of decisions made without the data are the wrong ones.

When promoting data-driven decision making at Microsoft we distributed 1000s of these little hippo squeeze toys.  Perhaps by squeezing our hippo toy, we are reminded to constrain our HiPPO from making rash decisions without data (somewhat look a voodoo doll).

But another, kinder way to look at the HiPPO is as the person who has final responsibility for product feature decisions, and as a reminder to get that person the data they need to make the data-driven decisions.

Either way, it reminds us that data trumps intuition.


[and remember the hippo is considered to be the most dangerous animal in Africa (not counting the mosquito)] :-)