How to Reduce MTTR by Shortening Diagnosis, Not Detection

MTTR, usually expanded as mean time to resolution or mean time to repair, measures how long a service stays broken once an incident starts. We care less about the exact expansion than about which part of that clock is long, because each part responds to a different fix.

Most advice on how to reduce MTTR targets detection and paging. Those are worth doing once, but after a team pages in minutes, the hours go somewhere else.

In this post we cover why faster alerting stops helping, how to split the incident clock into timed phases, six steps that shorten diagnosis, how to record diagnosis time, and when this approach will not help at all.

Why Faster Alerting Stops Reducing MTTR After a Point

AWS describes the gap plainly in its resilience whitepaper on reducing MTTR. "Part of the repair process after detection is the diagnosis period," it says, the time operators spend querying logs, reading health metrics and logging into hosts to work out what is wrong.

Detection improvements stop paying once a page arrives within a few minutes. Shaving another minute off alerting does little when the next hour goes to deciding which graph matters. We see the same pattern with more dashboards, which tell an engineer that something is broken but rarely why.

Runbooks help with the incidents a team has already had. They do much less for a novel failure several layers down, where the expensive part is working out what kind of incident it is.

The Visible Ops Handbook is often cited for the estimate that about 80 percent of MTTR goes to identifying what changed. We have not seen the underlying data, so we treat the figure as a reason to measure our own split rather than as a benchmark.

Split the Incident Clock Into Five Timed Phases

An undivided MTTR number hides where the time goes. We split every incident into five phases, each with a start and end timestamp that someone records during the incident, not afterwards from memory.

Phase	Starts when	Ends when	What usually shortens it
Detection	Impact begins	An alert fires or a user reports it	Alerts on user-facing symptoms
Acknowledgment	The alert fires	A responder accepts the page	Clear routing and a sane on-call load
Diagnosis	A responder starts looking	The cause is known well enough to act	Evidence in one place and a fixed search order
Mitigation	The team decides what to do	Impact stops	Rollbacks, feature flags and traffic shifts
Validation	The fix is in place	The team confirms impact has not returned	Checks defined before the fix ships

Two of these overlap in practice, because a good mitigation often lands before diagnosis ends. We record both timestamps anyway, since the gap between them shows how often a team stops the bleeding before it understands the cause.

How to Reduce MTTR in Six Steps That Target Diagnosis

These are the steps we follow, in this order, once detection and paging are reasonable. Each one removes a specific kind of wasted time from the diagnosis phase.

1. Mitigate First and Diagnose Second

The instinct during a major outage is to find the root cause. The Google SRE book's chapter on effective troubleshooting pushes the other way, saying "your course of action should be to make the system work as well as it can under the circumstances." We divert traffic, shed load or roll back first, then investigate a system that is no longer burning.

2. Ask What Changed Before Anything Else

Deploys, configuration pushes, feature flags, schema migrations and dependency upgrades come first in our search because they are cheap to check and usually easy to undo. The AWS whitepaper describes returning to a known good state as a common mitigation, by rolling back a recent change. DORA tracks the same idea as failed deployment recovery time, the time to recover from a deployment that needs immediate intervention.

The catch is that many changes never appear in the deploy log. Feature flag flips, configuration pushed from a separate repository, a managed database's maintenance window and a dependency's own release all change production without a deploy. We keep a list of those change sources next to the runbook, so the question "what changed" has a complete answer within the first few minutes.

3. Put Every Piece of Evidence on One Timeline

Much diagnosis time is spent switching between tools and lining up timestamps by hand. We build one timeline of deploys, error rates, log exceptions, slow queries and infrastructure events such as restarts and OOM kills. That correlation work is where an AI agent that investigates with evidence earns its place, because it can gather the timeline before a human finishes reading the page.

4. Test One Hypothesis at a Time

The SRE book frames troubleshooting as the hypothetico-deductive method, proposing a cause and testing it against observations. We write each hypothesis down with the evidence that would confirm or rule it out, and test the cheapest one first. A team that tests three ideas in parallel without writing them down usually tests none of them properly.

5. Check the Proposed Root Cause Independently

A deploy at 14:00 and an outage at 14:01 is a correlation, not a cause. Before acting on a diagnosis, we ask someone or something that did not produce it to test it, for example by checking connection pool levels against their normal baseline when the claim is a connection leak. This one step prevents the most expensive kind of MTTR, the second incident caused by fixing the wrong thing.

6. Hand Over the Fix as a Reviewable Change

Once the cause is confirmed, the fastest path to a lasting repair is a small, reviewed change with the evidence attached. We keep the reviewer in the loop on purpose, because a fix that skips review can reopen the incident.

The handover note should carry the timeline, the hypothesis that survived, the evidence that confirmed it and the check that will show the fix worked. A reviewer who gets all four can approve in minutes. A reviewer who gets only the diff has to repeat the investigation before trusting it.

Our guide to root cause analysis that does not stop at the first why goes further into steps 4 and 5.

Record Diagnosis Time in Every Postmortem

The steps above only stick if the team can see their effect. We add three fields to every postmortem, built from the phase timestamps:

• Diagnosis time, from the first responder starting work to the moment the cause was known well enough to act.
• Mitigation lag, from the cause being known to impact actually stopping.
• The search path, meaning the hypotheses tested and discarded along the way.

Someone has to write these down while the incident is running. On larger incidents we give the job to a named scribe, and on small ones the responder posts a one-line note in the incident channel at each phase change, which is enough to reconstruct the clock later.

The search path matters more than it looks, because it shows which checks wasted time, a point we make in why the postmortem should keep the search. Two related questions are covered elsewhere, setting a target in what a good MTTR is and reading the distribution in why average MTTR misleads.

When a Diagnosis-First Approach Will Not Shorten MTTR

This method assumes diagnosis is the long phase, and sometimes it is not. We check three situations before investing in it.

• Failures That Are Routed Around Automatically

The AWS whitepaper describes load balancer health checks that mark a target unhealthy within about ten seconds, at which point MTTR is effectively the same as detection time. If most incidents end that way, diagnosis is a post-incident activity and better failover pays more than faster investigation.

• Incidents Dominated by Repair Work

Restoring a large database or rebuilding lost state can take hours after the cause is obvious. For those, we invest in restore drills and smaller blast radii, not diagnosis.

• Too Few Incidents to Measure

A team with a handful of incidents a year cannot tell a real improvement from noise. We still record the phase timestamps, but we judge changes by individual postmortems rather than by the average.

Start Reducing MTTR With the Next Incident's Timestamps

We would begin with the next incident, recording five phase timestamps and the search path, then look at where the longest gap sits after a few incidents. If it is diagnosis, the six steps above apply in order.

That is how to reduce MTTR once alerting is already fast, by spending less time deciding what is wrong. Operate is built for that phase. It brings the evidence from logs, databases and code together through read-only adapters, and a separate agent on a different model checks each proposed root cause against that evidence.