Blameless Postmortems Record the Answer and Lose the Search

A blameless postmortem is a written review of an incident that looks for contributing causes without blaming individuals. The Google SRE book puts it plainly. To be blameless, a postmortem "must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior."

We think blamelessness was the right fight, and it was largely won. What it left untouched is the shape of the record. Most templates still capture what the system did and the answer the team reached, and very little of how they reached it.

In this post we cover what researchers saw when they watched real incident responses, what the same companies kept in their reviews, why the template looks the way it does, four fields that record the search, what changes once you have them, and when a short timeline is enough.

Two Records of the Same Incident

Take an illustrative database outage. The review says a configuration change deployed at 10:00, latency rose at 10:05, the change was reverted at 10:45 and the system recovered at 10:50. Root cause, a bad connection pool setting. It is accurate and complete as a history of the system.

The responder's hour looked different. A dashboard showed the database nodes green, so the first fifteen minutes went on a suspected network partition. Then the load balancer, then disk I/O, then an application change. A latency metric aggregated across services pointed at the wrong one for twenty minutes.

Only the first record usually survives. The second, the record of the search, is what explains why the incident took fifty minutes rather than five, and it is what the next responder would most want to read.

What the STELLA Report Observed About Anomaly Response

In 2017 the SNAFUcatchers consortium held a workshop on how software teams cope with complexity, and published the result as the STELLA report. Participants walked through real anomalies from real firms, and the report describes what responding to them involved.

Its summary of the search is the line we quote most often. "In every case the search was effortful and iterative. Neither the sources of the anomaly nor the route to correction were immediately apprehended." Responders explored the system through their own internal models of it, which the report places "below the line" of what anyone can observe directly.

We read that as a description of the work itself, not as a sign that the responders were slow. Forming hypotheses, testing them against ambiguous signals and revising the mental model is what incident response is.

What the Same Firms Kept in Their Postmortems

The same report then describes how those firms ran their reviews. Facilitators gathered machine logs and chat histories, interviewed key participants, and presented the event with "a skeletal timeline" in a meeting of about an hour, held days or weeks later. Attendees added technical details and "what were we thinking about" comments.

The report calls the resulting postmortems "prosaic to the point of being narrow", concentrated on technical details and the evolution of the event through recovery. It also notes that pressure to produce action items sometimes led to abbreviated timelines and limited the analysis.

So the search is spoken about in the room, at least in part. What we rarely see is a place in the written record that is designed to hold it, which means it fades with the memories of the people who were there.

Why the Blameless Postmortem Template Looks the Way It Does

We should be fair to the template. A timeline rebuilt from logs is objective, quick to assemble and hard to argue with. A record of three wrong hypotheses can read as a record of incompetence to someone who was not there, which is exactly what blameless practice tries to avoid.

Good templates already go further than a timeline. Google's SRE workbook example postmortem includes lessons learned, with what went poorly and where the team got lucky. Those sections capture judgement, but they are written with hindsight and rarely preserve the order of the search.

The result is a reasonable response to how hard it is to record thinking. We still think it is the wrong default for a team that wants its diagnosis to get faster.

Four Fields That Make a Blameless Postmortem Record the Search

We do not replace the existing process. We add four fields to the template and fill them during preparation, from interviews and chat logs, before the meeting starts and hindsight begins rewriting the story into a straight line. Each field answers a question a timeline of system events cannot.

• Which explanations did responders try, and what ruled each one out?
• Which signal pointed the wrong way, and for how long?
• What evidence did not exist that would have ended the search early?
• How long did it take before anyone proposed the right answer?

1. A Hypothesis Log

A short table of each hypothesis the responders formed, the evidence that suggested it, the evidence that ruled it out and roughly when. The wrong hypotheses are often the most useful rows, because they show what the system's signals made plausible at the time.

The order matters as much as the content. Which signal an experienced responder checked first, and why, is a heuristic that newer engineers cannot learn from a timeline of system events.

2. The Signal That Misled

Any dashboard, metric or alert that pointed the wrong way or looked fine while the system was failing. A green panel that should have been red, or an average that hid a tail, is a defect in the observability layer. In a timeline it is invisible. In this field it becomes a ticket.

3. The Instrument That Did Not Exist

The query nobody had written, the log field that was missing, the trace that stopped at a service boundary. We ask responders what single piece of evidence would have cut the search from forty minutes to four. These are usually the highest-value action items a review can produce.

4. Time to the First Correct Hypothesis

Record the time to first correct hypothesis, meaning when the right explanation was first proposed, next to detection and resolution times. The gap between the start of the incident and that moment is the cost of diagnosis, and it is the part of resolution time a team can most directly shrink, as our guide on reducing MTTR by shortening diagnosis argues.

Recording it also stops a review from ending at the first plausible answer. Our post on running root cause analysis without stopping at the first why covers the method side of the same problem.

Patterns Appear Once the Search Is on Record

A single hypothesis log is useful. A year of them is a dataset. If the same load balancer panel shows up as the signal that misled in three unrelated incidents, the team has found a systemic weakness that no timeline would ever surface.

The metric changes too. With time to the first correct hypothesis on every review, we can finally ask if diagnosis is getting faster, or if incidents are simply getting easier. Searching past reviews for similar searches is its own problem, which our essay on retrieving incident history takes up.

The same standard applies to automated investigation. A tool that investigates should leave a record of which signals it read and which hypotheses it ruled out, the way an AI agent that investigates with evidence should, so that its conclusion can be reviewed like a human one.

When a Short Timeline Is Enough

Not every incident needs a hypothesis log. When the cause was obvious from the first alert and the fix took minutes, the search was trivial and recording it adds nothing. We reserve the four fields for incidents where diagnosis took a meaningful share of the time.

The fields also depend on trust. If people fear that a list of wrong guesses will be used against them, they will leave it out or tidy it up, and a tidied search is worse than none. The blameless culture has to exist before this record can be honest.

Finally, interviews cost time. A facilitator who spends an hour reconstructing the search for a minor incident is spending it badly, so we scale the effort to the incident's severity and to how surprising it was.

Make the Next Blameless Postmortem Keep the Search

The industry spent a decade making postmortems safe. We would spend the next step making them useful for diagnosis, by giving every blameless postmortem a place for the hypotheses, the misleading signals, the missing instruments and the time it took to find the right answer.

Add the four fields to your template before the next review. Operate runs inside your own infrastructure, and a separate agent on a different model checks each proposed root cause against its evidence before an engineer reviews it.