Production incidents on a customer's systems: assign roles first, fix second, and write the postmortem so it doesn't happen again

Picture it: 9pm, and the shared chat channel with your customer suddenly fills with messages. The data sync job you deployed last week is overwriting orders. Three people on the customer’s side are asking questions at once, one of their engineers has already logged into the server to “try a restart”, and you have the logs open while answering the phone.

At that moment, writing good code is not enough to rescue the situation. What decides the outcome is whether you know how to coordinate: who may touch the system, who talks to the customer, who records what happened.

For an FDE, incidents rarely happen “in house”. You stand between two organisations, two engineering teams and two sets of processes. How you handle the first hour, and the postmortem you write afterwards, will determine how far the customer still trusts you.

Why declare an incident early?

Google’s SRE book advises declaring an incident early rather than waiting until the problem has grown over several hours. Engineers’ instinct usually runs the other way: fix it quietly first and report afterwards, for fear of making a fuss.

At a customer, the cost of reporting late is even higher. If the customer spots the problem before you do, they will act on their own terms, and the story becomes “the vendor hid the bug”. Declaring early and downgrading later is far cheaper than declaring late.

Declaring needs no ceremony. A clear message in the shared channel is enough:

[INCIDENT] The order sync job has been overwriting records since 20:50.
Impact: some of today's orders may have the wrong status.
Commander: Minh (FDE). All system changes go through Minh.
Next update: 21:30, or sooner if there is progress.

The third line matters most, because it immediately stops everyone from applying their own fix.

Four hats, even with only two people

PagerDuty describes the Incident Commander as the single place that holds accurate information about what is going on. Google adds a hard rule: while the incident is under way, only the operations team may change the system, while the commander keeps the overall picture.

The other two roles are often forgotten. PagerDuty defines the Scribe as the person who records the timeline during the incident, and the customer liaison as the person who talks directly to the customer. Google’s book calls a similar role the communications lead, the person who speaks for the whole response team.

For small incidents, PagerDuty allows one person to hold several roles. In practice at a customer, the FDE is often both commander and liaison. What must be avoided is a role that nobody holds.

Role	Who holds it in the example	Main job	Does not
Incident Commander	FDE	Makes decisions, keeps the shared status	Type fix commands themselves
Operations	A designated engineer	Carries out agreed changes	Experiment outside the plan
Scribe	A colleague or the FDE	Records a timeline with timestamps	Analyse the cause mid-incident
Customer liaison	FDE or account lead	Updates at the promised times	Promise a time for the fix

With the customer’s engineers, this is the hardest part to say. You have no authority to give them orders, but you can propose it directly: “Could you take the operations role? Let’s agree every command in this channel before running it.” Usually they are relieved that someone has taken the wheel.

The first hour, seen through the timeline

Continuing the hypothetical example above, the timeline the Scribe records might look like this. Note that each line records an event, not a judgement.

20:50  Sync job runs on schedule.
21:04  Customer reports wrong order statuses.
21:07  Minh declares the incident, assigns roles.
21:12  Operations temporarily disables the sync job via a config flag.
21:20  Confirmed: no new overwrites.
21:30  Customer update: spread contained, assessing the scope of affected data.

22:15  Cause identified: a key-mapping change in the latest deploy.

The order is deliberate: contain the damage first, find the cause later. At 21:12 nobody yet knew why the job had failed, but switching it off was enough to stop data being written incorrectly.

If the incident runs into the next day, Google stresses that command must be handed over explicitly at the end of the working day. A line such as “From 8am, Lan is commander; the timeline is at this link” avoids two people each believing they are at the wheel, or nobody being there at all.

Postmortems: fix the system, not the people

Google defines a postmortem as a written record of an incident, its impact and the actions taken to mitigate or resolve it. The reason to write one is simple: according to the SRE book, without a formal process for learning from incidents, they can recur indefinitely.

When is one mandatory? Google advises setting trigger criteria in advance, one example being data loss of any kind. The overwritten orders in the example fall squarely under this criterion, so nobody has to argue about whether to write one.

The spirit of the document is blameless. The SRE book says you cannot “fix” people, but you can fix systems and processes to better support them. “Minh deployed the wrong mapping” teaches nothing. “The deploy pipeline does not check key mappings against real data” points to the work that needs doing.

Cloudflare’s public postmortem of its 18 November 2025 incident is a model worth studying. It opens with a direct apology from leadership: on behalf of the whole team, they apologised for the harm caused to the Internet that day. The trigger is stated plainly: a permissions change on a database system.

The list of remediations is equally concrete, including adding more global kill switches for features.

The kill-switch detail is worth an FDE remembering. In this article’s example, the 21:12 step was only quick because the job already had an off flag. Without one, that is exactly the first item that belongs in the postmortem’s action list.

A postmortem sent to a customer can be a single page: a summary and apology, the impact (how many records, over what period), the timeline, the trigger, what was done to resolve it, and preventive action items, each with an owner and a deadline.

A small exercise to build the blameless reflex: take the line “the customer’s engineer logged into the server and restarted it” from the opening scenario and rewrite it as a flaw in the system.

Hint: ask what made that action possible and seemingly reasonable at the time, for instance the lack of a channel for agreeing commands, or nobody having been assigned the operations role. Your answer is a preventive action item.

What if the customer has never written a postmortem?

This is common. Google’s SRE Workbook acknowledges that introducing postmortems into an organisation is as much a cultural change as a technical one. Its advice is to start small with a very basic process and adapt it gradually to fit the organisation.

For an FDE, starting small means writing the first one yourself and inviting the customer to read and comment, rather than proposing an entire new process. Once the customer sees a document that blames no one yet produces concrete work, next time they will ask whether you are writing one.

Common mistakes

The most common mistake is letting several people fix things at once. The customer’s engineer restarts a service while you are rolling back, and now nobody knows which change had any effect. The second is having no Scribe: when it comes to the postmortem, the whole team has to reconstruct the timeline from memory and chat history.

The third is in communication: promising a fix time before the cause is known. Promise only the time of the next update, then keep that promise. The last is writing the postmortem and filing it away without assigning any action item to an owner, so the old incident comes back exactly as the SRE book warns.

If you are aiming for an FDE role, read closely the job descriptions that mention on-call, incident response or working directly with customer infrastructure. On your CV, don’t just write “handled production incidents”. Spell out that you commanded incidents, how quickly you contained the damage, and which postmortems and preventive action items were implemented.

Employers need to see that you can coordinate, not just read logs.

Every system you deploy will have an incident sooner or later. What the customer remembers afterwards is who took the wheel at 9pm, and which document stopped that failure happening a second time.