Reliability Reporting Cannot Detect a Ten Percent Gain
A Google simulation on real incident data shows a genuine 10 percent MTTR gain goes unseen up to 40 percent of the time, and noise can look like a win.
Quick Answer
Reliability reporting built on average incident duration cannot reliably detect a real ten percent improvement. In a Google simulation on public incident data, a genuine 10% gain showed up as a worsening in 20% to 40% of runs, and no change at all produced an apparent half-hour win in 10% to 23% of runs. Count one named failure class at a time, and report uncertainty with every number.
Every engineering leader eventually faces the quarterly question, which is if the reliability work paid off. Most reach for a dashboard of incident metrics, namely mean time to recovery, incident count and severity mix, and look for a line that points down.
We think those inputs cannot answer the question. The problem is not only that the metrics are shallow. It is statistical power, since at the sample sizes and variance found in most organisations, aggregate incident metrics cannot detect the change you are trying to measure.
The Quarterly Question Leaders Cannot Answer With MTTR
Incident data is low in volume and high in variance. A company with a few hundred incidents a year has few observations, and a handful of long outages dominate any average. Reporting a trend from that is closer to reading tea leaves than to measuring.
This post covers the simulation that shows the problem, why durations behave this way, what duration does not tell you, what has enough power to detect a change and how we would report honestly upward.
What the Monte Carlo Simulation Found
The strongest evidence comes from Štěpán Davidovič's O'Reilly report Incident Metrics in SRE, published through Google. He drew two equal samples from real incident data sets, shortened the incidents in one by 10 percent, compared the mean time to recovery of the two and repeated the process 100,000 times.
We use the report's own figures here, and we have not rerun the simulation. The report anonymises three public data sets as Company A, B and C, with 173, 103 and 609 incidents in 2019.
| Scenario | Company A | Company B | Company C |
|---|---|---|---|
| Real 10% improvement, MTTR difference fell below zero | 38% | 40% | 20% |
| Real 10% improvement, chance of seeing at least 15 minutes gained | 49% | 50% | 64% |
| No change at all, chance of seeing a half-hour improvement or better | 19% | 23% | 10% |
The report puts the consequence plainly. "If you can't tell when things aren't changing, you'll have a hard time telling when they do." In its words, a product that did nothing would still look like a purchase worth making in a meaningful share of simulated universes.
Why Incident Durations Behave This Way
The report finds that incident durations follow a positively skewed distribution in every data set it examined. Most incidents resolve quickly, a few linger and a handful are disastrous. We read that as the root of the power problem, because a single complex outage can swamp a quarter of small improvements.
Switching from the mean to the median is often suggested as a fix. The report examines it and finds that the difficulty is not specific to the mean in MTTR, and that median time to recover has the same trouble. Our analysis of median versus mean in outage data looks at the tail in more detail.
Duration Does Not Stand In for Severity
If duration is noisy, leaders often fall back on severity and track MTTR only for the worst incidents. Courtney Nash's article on the VOID in InfoQ reports that "There is no correlation detected between incident duration and incident severity." Long incidents can be minor and short ones can be existentially critical.
The same article calls duration "gray data", which is high in variability and low in fidelity. We take from it that closing a ticket quickly does not show the fire was small, and that filtering by severity does not rescue the metric. The VOID report is where the underlying dataset lives.
What This Does to the Budget Conversation
The case for a platform hire, observability spend or new SRE tooling is often made with an instrument that cannot detect the effect it is asked to prove. We see the cost in both directions.
- Invisible wins. Effective programs get defunded because MTTR did not move.
- Phantom wins. Ineffective tools get scaled because a random cluster of short incidents created a temporary downward trend.
This is a measurement design problem, not a culture problem. The way out is to stop asking the data to do something it cannot do.
What Has Enough Power to Detect a Change
To gain power you need more observations and less variance. That points away from portfolio-wide averages and toward specific classes of failure.
- Change the unit from duration to count. Counts of one named failure class, such as out-of-memory kills in the checkout service, have lower variance and higher frequency than the durations of all incidents combined.
- Pick high-frequency events. Look for effects that occur hundreds of times a quarter, such as one error signature, pages from one noisy alert rule or deployments that needed a manual rollback.
- Use a targeted before and after. If a fix addresses one bug class, measure only that class, because the counterfactual is far easier to read than a global average.
The obstacle is that attributing failures to a class is usually manual and inconsistent. That attribution work is where we would let dedicated AI agents that investigate with cited evidence help, while people confirm the class. Our piece on why recurrence rate tells you what closure rate hides shows how class-level counts look in practice.
Honest Reliability Reporting Needs Intervals and Classes
When reporting to leadership, we would move from point estimates to intervals. If a 12% MTTR improvement comes with an interval that spans zero, say so. Present class-based counts beside the aggregate metrics instead of trying to replace the aggregates in one move.
A leader who says that one class of failure fell from 340 occurrences to 40 after work on the ingestion pipeline is making a claim that survives scrutiny. A leader who says MTTR improved by 12% is reporting something the data cannot support. The InfoQ article recommends alternatives such as service level objectives, customer feedback, post-incident reviews and near-miss analysis over another single metric.
What Not to Conclude From This Power Problem
This is not an argument that incident duration is irrelevant, or that you should stop measuring. Speed of recovery matters to customers, and a single very long incident deserves a review whatever the average says.
It is an argument against one specific claim, that our work improved reliability by a stated percentage as shown by aggregate MTTR at a normal company scale. We would still track duration, and we would stop using it to prove effect. For targets, our guide to setting your own MTTR target covers what the number can legitimately support.
Stop Measuring the Clock and Start Measuring the Class
Reliability reporting improves when the unit changes from the clock to the class. If you want to prove that reliability work matters, pick a failure class, count it before and after, and report the uncertainty. Aggregate MTTR can still describe a quarter, and it cannot reliably tell you if your work changed it.
We would pick one class this quarter and count it weekly. When you want help attributing causes to classes with evidence, Operate runs dedicated AI agents that investigate production problems with cited evidence and propose fixes for your team to review.
Frequently Asked Questions
We think it is a weak basis for decisions. Davidovič's simulations on real incident data found that mean time to recovery misses real ten percent gains and shows false ones. It can describe a period, and we would not use it as proof that a change worked.
We point to skew and small samples. Incident durations are positively skewed, so a few long outages dominate the mean, and most organisations have too few incidents for the average to settle. The result is a number that moves for reasons unrelated to your work.
We would count a single named failure class, which has higher frequency and lower variance. Pair it with service level objectives, customer feedback and post-incident reviews, as the VOID analysis suggests, so that you have evidence of impact beyond the clock.
Not according to the VOID analysis, which reports no correlation between duration and severity. We take that to mean long incidents can be minor and short ones can be critical, so filtering by severity does not rescue duration as a measure.
Not enough to solve the problem. The Google report found the difficulty is not specific to the mean and that median recovery time is also hard to measure reliably. We use medians for description and class counts for judging whether work paid off.
About the author
Operate Team
The team behind Operate
Operate Team builds Operate, enterprise AI agents for production operations. Operate answers production questions from Slack and Microsoft Teams, investigates alerts, and reviews performance, security and cost. It finds the root cause with evidence and proposes a fix for an engineer to review, with controlled access to your systems.
