Production Incidents: One of My Favorite Things | Ramblin' & Amblin'
Reflections on production incidents and system outages: how acute crisis focuses senior executive leadership, unites cross-functional engineering teams, and creates fleeting windows for meaningful architecture improvements before business as usual returns.
The Interviewer
Mike Hall
Host, UGtastic / Ramblin' & Amblin'
Mike Hall
Host, UGtastic / Ramblin' & Amblin'
▶ Speaking
One of my favorite things about production incidents was it had a focusing effect on a large number of people in senior leadership positions that was breathtaking sometimes in its acuity. Seeing people that had argued against every type of maintenance or support all of a sudden be very focused on maintenance, support, and getting a fast resolution to a problem. Not to be cynical, but people losing focus is a big thing. Production incidents, like all good emergencies, are moments not to waste that opportunity to focus people and raise up issues around maintenance and the value of maintenance when it gets forgotten.
Mike Hall
Host, UGtastic / Ramblin' & Amblin'
▶ Speaking
Whether it is that you physically cannot do it, you do not have resources or money, or that a narrative has been sold to executives who signed off on a big budget and you have to deliver on that promise. Trying to get people to be aware of maintenance during the happy times is difficult. But when production is down, all of a sudden everybody comes together.
Mike Hall
Host, UGtastic / Ramblin' & Amblin'
▶ Speaking
There is this scene from the movie The Man from U.N.C.L.E. where Henry Cavill is going into a fight, hikes up his sleeves in an old school boxer pose, and prepares for action. That was based off me preparing for a production incident where I had laid out a map for how everything works so I was able to leverage that to great effect. In those moments, what was good is being able to point out that we can come together, fix things, come to a better understanding of the systems we are responsible for, and have meaningful dialogue.
Mike Hall
Host, UGtastic / Ramblin' & Amblin'
▶ Speaking
When you are all split up across different teams with different agendas driven by non-technical priorities, it is hard to get people back together. Nothing in my experience quite brought people together for even a few stressful moments than a production incident. The beauty of reporting MTTR is that then we have a standardized template for reporting how well we handled an incident, which did nothing to fix the underlying problem before going back to business as usual.
Mike Hall
Host, UGtastic / Ramblin' & Amblin'
▶ Speaking
In those moments where everybody else was dreading it, I was having a good time because usually I was the one who knew what was up, because I had spent a lot of time mapping it out and understanding the domain and the machine I was responsible for. Being able to help everybody else stay calm in chaos is a product of the environment. I think about Robert De Niro's character in Brazil, the rogue heating engineer who would go from unit to unit fixing whatever needed to be done while the system was mad at him for interrupting their bureaucratic processes.
Mike Hall
Host, UGtastic / Ramblin' & Amblin'
▶ Speaking
When you can tap someone and get them to be part of the solution, pulling people from other departments using Cynefin, sense making, and triage response strategies, they went from being stressful to creating a lasting sense of relief and camaraderie. That is one of the things I always loved about being a software engineer: those moments where intelligent people come together, collaborate, and solve a problem. It is addictive. Alright, thanks, bye.