The first thirty minutes of something breaking
The instinct when something breaks is to find out why. That is the second job. The first is to stop it costing anything.
Everything breaks eventually, and the difference between a bad hour and a bad week is almost entirely decided in the first half hour, before anyone knows what happened.
Stop the bleeding before you understand it
Curiosity is the enemy here. Understanding the root cause is deeply satisfying and can take three hours, during which customers keep hitting the same broken thing.
Take the fastest action that reduces harm, even an ugly one: roll back, disable the feature, put up a notice, switch to the manual process. None of those are fixes and all of them buy you the calm you need to think. The cause will still be there afterwards, and you will diagnose it better without an audience.
One person coordinates, and says so
Three capable people all investigating independently is worse than one, because they duplicate work, contradict each other in the same thread, and nobody is watching whether the situation is improving.
Name a coordinator out loud — "I am running this" — even informally. Their job is not to fix it; it is to decide who does what, keep everyone else out of the way, and be the one who talks to the rest of the business. If you are the most senior person present, this is your job, not the debugging.
Write it down while it happens
Nobody reconstructs an incident correctly afterwards. Times compress, the order of events rearranges, and the thing that fixed it gets confused with the thing you tried just before it.
Keep a running note as you go: the time, what you observed, what you changed. Two lines each. It costs nothing during and it is the entire difference between a post-mortem that finds something and one where four people argue from memory about what happened at 14:20.
Say something before you have the answer
Teams stay quiet during an incident because they are waiting to have something useful to say. Meanwhile support is guessing, and customers are filling the silence with their own explanation, which is always worse than yours.
Send a short message early: what is affected, what you are doing, when you will update next. Then actually update at that time, even if the update is that there is nothing new. Predictability during an outage buys more goodwill than speed does.
Questions people actually ask
Should I find the cause or fix the symptom first?
Stabilise first, even with an ugly action — roll back, disable the feature, switch to a manual process. The cause will still be there afterwards, and you will diagnose it better without customers hitting the same break.
Who should be in charge during an incident?
One named coordinator, said out loud. Their job is not fixing it but deciding who does what, keeping others out of the way, and talking to the rest of the business. If you are the most senior person present, that is your role.
When should we tell people?
Before you have the answer. Send what is affected, what you are doing, and when you will update next — then update at that time even if there is nothing new. Predictability buys more goodwill than speed.
