You Cannot Detect a Ten Percent Improvement: The Power Problem in Reliability Reporting

TL;DR: Statistics show that a 10% real improvement in incident response is invisible up to 40% of the time, while pure noise looks like a win up to 23% of the time. Because incident data is low-volume and high-variance, aggregate metrics like MTTR lack the statistical power to prove whether your reliability investments actually worked.

The quarterly question

Every engineering leader eventually faces the same high-stakes moment: a quarterly business review or a board meeting where they must answer, "Did our reliability work pay off?"

To answer, most leaders reach for a standard dashboard of incident metrics: Mean Time to Recovery (MTTR), total incident count, and the severity mix. They look for a downward trend in the line and present it as proof of efficacy.

This article argues that these inputs cannot answer the question. The problem isn't just that these metrics are "shallow" or culturally reductive—it is a fundamental problem of statistical power. At the sample sizes and variance levels found in almost every engineering organization, aggregate incident metrics do not have the power to detect the changes you are trying to measure.

The simulation, in full

The most damning evidence against these metrics comes from research conducted by Stepan Davidovic, published by Google and O’Reilly in Incident Metrics in SRE. Davidovic performed a Monte Carlo simulation using real incident data from three different companies. He created two equally sized samples for each: one untouched (the control) and one with a 10% experimental improvement applied (the treatment). He then ran 100,000 simulated comparisons to see if the metric would actually catch the win.

The results, cited extensively in the VOID reports, should change how every CTO views their dashboard:

According to Davidovic’s research, when a real improvement was applied, the MTTR difference still came out negative (showing things got worse) in up to 40% of runs. Even worse, when he ran the simulation with no change at all, between 10% and 23% of runs showed an apparent improvement.

Key Takeaway: Aggregate incident metrics miss real gains roughly a third of the time and manufacture fake wins roughly a fifth of the time, making them statistically indistinguishable from a coin flip.

As Davidovic concluded, if you cannot tell when things are not changing, you will have a hard time telling when they do.

Why the data behaves this way

The reason incident metrics fail is that incident durations are heavily "positively skewed." In a typical system, most incidents resolve quickly—perhaps within 30 to 60 minutes. However, the "tail" of the distribution is long and thin, containing the six-hour or twelve-hour outliers that drag the mean upward.

This skew, combined with a small sample size ($n$), creates the power problem. A single complex outage can obliterate a quarter’s worth of minor improvements. While switching to the median is often suggested as a fix, Davidovic’s simulations showed the same failure rates for the median, geometric mean, and absolute duration sum.

Furthermore, switching from mean to median can make your reporting look 2.5 hours "better" simply by changing the calculation method, without any change in the underlying system. A metric that improves because you changed the math is not a metric; it is a narrative tool.

The second finding, which removes the fallback

If duration is unreliable, leaders often fall back on "severity" to filter the noise—focusing only on MTTR for SEV-1s. However, the Verica Open Incident Data (VOID) report, which analyzed over 10,000 incidents across hundreds of companies, found no correlation between incident duration and incident severity.

According to the VOID report, long incidents are frequently trivial, while critical ones can resolve in minutes. In a study of Honeycomb’s incident data, three of the five longest incidents had minor or no customer impact, while two of the three most critical incidents were resolved in 21 and 79 minutes respectively. Duration cannot stand in for impact, and closing a ticket quickly does not mean the fire was small.

What this does to the budget conversation

The practical consequence of this power problem is that the case for the platform hire, the observability spend, or the new SRE tooling is being made with an instrument that cannot detect the effect it is being asked to prove.

This costs the organization in both directions:

1. Invisible wins lead to effective programs being canceled or defunded because the "MTTR didn't move."
2. Phantom wins lead to the scaling of ineffective tools because a random cluster of short incidents created a temporary downward trend.

This is a measurement-design problem, not a culture problem. To fix it, we must stop asking the data to do something it is mathematically incapable of doing.

What has enough power to detect

To gain statistical power, you must increase $n$ and decrease variance. This requires moving away from portfolio-wide averages and toward specific classes of failure.

• Change the unit from duration to count: Counts of a single named failure class (e.g., "OOM kills in the checkout service") have much lower variance and higher frequency than the durations of all incidents combined.
• Pick high-frequency events: Look for effects you see hundreds of times a quarter, not tens. Examples include:
  • Occurrences of a specific error signature.
  • Pages for a single, noisy alert rule.
  • Deployments that required a manual rollback.
• Use targeted "before-and-after": If you implement a fix for a specific bug class, measure only that class. The counterfactual—what would have happened without the fix—is far more legible than a global average.
• Track cost-of-coordination: According to Laura Maguire’s research on the Costs of Coordination, signals like the number of people involved, teams paged, and concurrent incidents provide a more honest view of system stress than the clock.

How to report it honestly upward

When reporting to leadership, shift the focus from point estimates to intervals. If you report an MTTR "improvement" of 12%, but the statistical interval spans zero, you should say so.

Courtney Nash, lead researcher for the VOID, suggests a practical transition: present deeper, class-based data alongside the expected aggregate metrics rather than trying to replace them in one move. A leader who says, "This specific class of failure went from 340 occurrences to 40 because of our work on the ingestion pipeline," is making a claim that survives scrutiny. A leader who says "MTTR improved 12%" is just reporting noise.

The reason class-level counts are rarely available is that attributing failures to a class is traditionally manual and inconsistent. Operate solves this by automatically opening cases for recurring failure classes—such as slow queries, flaky CI, or latency regressions—and attributing a verified cause to each. By creating a countable record of specific failure classes, it provides the high-powered data engineering leaders need to justify their work without relying on the coin flip of aggregate MTTR.

What not to conclude

This is not an argument that incident duration is irrelevant or that you should stop measuring reliability altogether. Speed of recovery is a vital component of resilience.

It is, however, an argument that aggregate incident metrics at a normal company scale do not support the claim: "Our work improved reliability by X percent." If you want to prove your work matters, stop measuring the clock and start measuring the class.

Sources & further reading

• According to Verica, the Monte Carlo simulations prove that MTTR is an unreliable indicator of organizational performance.
• According to the VOID Report, there is no statistical correlation between how long an incident lasts and how severe it is.
• According to Courtney Nash, engineering leaders should supplement aggregate data with "gray data" and qualitative analysis to understand system resilience.
• [Stepan Davidovic, Incident Metrics in SRE, Google/O'Reilly] provides the underlying statistical foundation for the power problem in reliability reporting.