Second Opinions in Root Cause Analysis and the 21% Problem

Second opinions in root cause analysis are independent reviews of an incident's diagnosis by someone who did not produce it, working from the original evidence. We think they are the most underused safeguard in incident response, and the best evidence for them comes from medicine rather than software.

Clinicians have measured how often a first diagnosis survives a fresh look. Engineering teams mostly have not, so we borrow their numbers with care and say where the comparison stops holding.

In this post we cover what the Mayo Clinic referral study found, why first diagnoses fail, why AI diagnoses make the problem harder, which incidents deserve a second read, and the limits of the medical comparison.

What the Mayo Clinic Referral Study Actually Found

The study behind the 21% figure is Extent of diagnostic agreement among medical referrals, published in the Journal of Evaluation in Clinical Practice in 2017 by Van Such, Lohr, Beckman and Naessens. It was a retrospective study at a single academic medical center.

The researchers took 286 patients referred from primary care between January 2009 and December 2010, and compared each referral diagnosis with the final diagnosis. Each case went into one of three categories.

• The referral and final diagnosis were the same in 12% of cases (36 of 286).
• The final diagnosis was better defined or refined in 66% of cases (188 of 286).
• The final diagnosis was distinctly different in 21% of cases (62 of 286).

The study also found that total costs were significantly higher for the cases with a different final diagnosis. We find that detail as useful as the headline, because it shows a wrong first diagnosis costs more than the time spent correcting it.

Why Second Opinions in Root Cause Analysis Change the Answer

The Mayo numbers do not say doctors are careless. They say a first diagnosis made under uncertainty, with partial information, is often incomplete, and that a second look from someone starting fresh changes it more often than people expect.

Incident response creates the same conditions. The first responder works under time pressure, with partial telemetry, and the moment a plausible story appears the pressure to restore service turns it into the root cause. We see three cognitive shortcuts doing most of the damage.

• Premature Closure Ends the Search Early

A study of 100 diagnostic errors in internal medicine by Graber, Franklin and Gordon found that cognitive factors contributed in 74% of cases. Premature closure, "the failure to continue considering reasonable alternatives after an initial diagnosis was reached", was the single most common cause.

In an incident channel this looks like a CPU spike that lines up with the outage. Someone declares resource exhaustion, and nobody asks why the CPU spiked or whether the spike was a symptom.

• Anchoring Holds the Team to the First Clue

Anchoring is relying too heavily on the first piece of information. In our experience the first clue is usually the recent deploys list, so a service that starts failing three minutes after a frontend release gets blamed on that release.

Later evidence pointing elsewhere is then read as noise. Teams can spend an hour reverting a clean build because they cannot let go of the first correlation.

• Availability Recycles Last Week's Incident

We diagnose with what we have seen recently. A team that fought a memory leak in the billing service last week will read this week's latency spike as another leak, and the underlying flaw stays hidden behind a familiar story.

Put the three together and a typical misdiagnosis follows. A database migration ran an hour before the outage, lock waits spiked, and the postmortem says the migration caused lock contention. The system recovers, perhaps because traffic dropped, and the incident closes.

If the real cause was a retry storm from a client hammering the database at the same moment, the migration tool gets fixed and the retry logic stays broken. We cover how to measure that kind of amplification in our guide to diagnosing a retry storm. A second reader who starts from the raw lock and connection data, without reading the first write-up, is the person most likely to notice the retry traffic.

This is why we treat the first root cause as a hypothesis until someone independent has tried to break it. The habit of asking why more than once helps too, and we describe it in running root cause analysis without stopping at the first why.

Automation Bias Makes AI Diagnoses Harder to Question

AI tools now propose root causes in many incident workflows, and that raises the stakes of the second read. A diagnosis that arrives from a machine tends to be questioned less than one that arrives from a colleague.

A systematic review of automation bias in clinical decision support by Goddard, Roudsari and Wyatt retrieved 13,821 papers and included 74. It found that over-reliance grew with workload, task complexity and time pressure, which describes an incident bridge almost exactly.

The same review lists what reduced the effect, including training, emphasising the user's own accountability, showing updated confidence levels with the advice, and presenting information rather than a bare recommendation. We read that as a design brief for any AI SRE. An AI agent that investigates with evidence shows the logs and queries behind its answer, so the engineer can check the reasoning instead of accepting a verdict.

Self-review does not solve this. A model asked to check its own diagnosis rereads the same context and tends to confirm it, which is why we argue the checker must be separate. We lay out that procedure step by step in verifying AI root cause analysis the way pilots cross-check.

Which Incidents Deserve an Independent Second Read

A second read costs an engineer's time, so we do not ask for one on every ticket. We reserve it for incidents where a wrong diagnosis would be expensive, permanent or likely to recur.

Incident type	Second read	Why
Severity 1 or customer-facing outage	Always	The fix and the postmortem will shape months of work
Recurrence of an incident marked as fixed	Always	The first diagnosis has already been contradicted by events
Root cause proposed by an AI tool	Before any production change	Automation bias lowers the scrutiny the answer gets
Diagnosis that drives an irreversible change	Always	Data migrations and vendor changes are hard to undo
Low-impact issue with an obvious, confirmed cause	Optional	The cost of the review exceeds the cost of being wrong

What makes the read independent matters more than who does it. We look for three properties.

• The reviewer was not on the incident and has not read the draft root cause before forming a view.
• The reviewer starts from logs, traces, metrics and the timeline, not from the summary.
• The postmortem records the alternatives the team ruled out and the evidence that ruled them out.

The last point is the one teams skip most often. A record of disproven hypotheses also helps future readers, a theme we take further in postmortems that record the discarded search.

Where the Medical Analogy Stops Holding

We use the 21% figure as a warning, not as an error rate for software. The Mayo patients were referred because their problems were uncertain, so the sample was selected for hard cases.

Other populations show lower rates. A study of 173 second-opinion patients at a Dutch academic hospital, Outcomes of second opinions in general internal medicine, found a new diagnosis in 13% of them. The rate of change depends heavily on how uncertain the first diagnosis was.

Incidents also differ from patients in ways that cut both ways. Production systems leave far more machine-readable evidence than a body does, which makes a second read cheaper and faster. They also keep changing during the investigation, so a reviewer working an hour later may be looking at a different system.

Finally, a second reader from the same team shares much of the first reader's context and habits. We get the most from reviewers who know the stack but did not build the component in question.

Make Second Opinions in Root Cause Analysis a Default for Severe Incidents

Second opinions in root cause analysis do not need a new process or a new team. We would start small. Pick the next Severity 1 postmortem, assign one engineer who was not on the incident, give them the raw evidence before the draft, and ask them to try to break the root cause.

If the answer survives, the team gains confidence it can defend.

If it changes, the team has found its own version of the 21% problem and a reason to make the second read routine. An independent check is built into how Operate investigates. A separate agent on a different model checks each proposed root cause against the cited evidence, and an engineer still reviews every finding.