Google found 84% of CI pass-to-fail transitions are flakes. Retrying hides the race conditions and shared state that later cause production incidents.

Retries Are Eating Your Signal: Flaky Tests Are System Bugs Wearing a Costume

TL;DR: Flaky tests are not just "test noise"—they are signals of system non-determinism like race conditions and shared state that will eventually manifest as production incidents. Automating retries in CI/CD pipelines effectively deletes the earliest and cheapest warning your system produces, trading long-term reliability for a temporary green build.

The engineering community has long treated flaky tests as a QA tax—a nuisance to be managed with quarantine lists and auto-retry logic. However, a recent debate ignited by a senior SDET on LinkedIn has reframed this perspective: flaky tests are not a testing problem; they are a system design problem. When a test passes four times and fails on the fifth without a code change, it is revealing a non-deterministic execution path that you did not account for.

By silencing that signal with a "retry" button, you aren't fixing the pipeline. You are muting a free chaos-engineering report.

The Flaky Test Data: A Warning from Google

The scale of this issue is larger than most teams realize. According to research from Google, approximately 16% of their entire test suite exhibits some level of flakiness. Even more startling is the transition data: Google found that 84% of pass-to-fail transitions in post-submit CI are flakes, rather than actual regressions.

If the vast majority of your Red builds are actually "non-deterministic noise," the natural reflex is to implement flaky test retries. But if 16% of your system is behaving unpredictably in a controlled CI environment, that same unpredictability exists in production, where the environment is significantly more volatile.

Why Retry-on-Failure is Signal Destruction, Not Stabilization

When you implement an automatic retry on a failed test, you are performing data erasure. You are telling your CI system: "I don't care why this failed; I only care that it can pass."

In doing so, you lose the "first-failure" data—the specific timing, state, and environmental conditions that allowed a latent bug to surface. Retrying hides the race conditions and shared state that later cause production incidents. If a test fails once in ten runs, you haven't found a "flaky test"; you've found a race condition that has a 10% probability of crashing a process or corrupting data in your live environment.

The Four Flake Classes and the Incidents They Predict

To treat flakes as system bugs, we must categorize them by the architectural debt they reveal.

1. Shared State → Data Races in Prod

Tests that fail because they depend on a specific database state or global variable are warnings. If your tests can't handle concurrent execution or unexpected state, your production services likely can't either, leading to hard-to-debug data corruption.

2. Timing Assumptions → Timeout Cascades

Hard-coded sleep() calls or tight timeout windows are the hallmark of why are my tests flaky. In production, these "timing assumptions" turn into timeout cascades when a downstream service experiences a slight latency spike, causing total system collapse.

3. Order Dependence → Deploy-Order Incidents

When Test B only passes if Test A ran first, you have identified a hidden dependency. In a microservices architecture, this manifests as a "deploy-order failure," where a new service version fails to start because it expects a specific handshake that is no longer guaranteed.

4. Infrastructure Flakiness → Noisy-Neighbor Problems

Sometimes the flake is external—a slow network or a resource-constrained CI runner. This is a direct preview of production "noisy-neighbor" issues. If your code can't survive a jittery network in CI, it will fail during a cloud provider's transient network event.

Key Takeaway: A flaky test is a pre-production incident report that identifies non-determinism before it reaches your customers.

A Triage Protocol: Treat Each Flake as a Miniature Incident

Instead of retrying, engineering leaders should adopt a "Flake-as-Incident" protocol:

Capture Evidence: Save the logs, heap dumps, and container state of the failed run immediately.
Identify the Class: Is this a race condition, a shared state issue, or a resource leak?
Root Cause Analysis: Use the evidence to find the non-deterministic path through the code.
Verify with Failure: Don't just fix it; prove you can trigger the failure reliably under specific conditions.
Remediate: Fix the system design, not just the test.

What to Measure Instead of Pass Rate

The standard "Pass Rate" metric is vanity. To understand CI health, measure:

Transition Rate: How often does a test flip from pass to fail without a code change?
Flake Half-Life: How long does a known flake persist in the codebase before a structural fix?
Retry Budget Burn: What percentage of your total compute cost is spent re-running failed tests?

When Retries ARE the Right Call

To be fair, there is a place for retries. In legacy systems with deep technical debt or when relying on third-party APIs beyond your control (like a sandbox payment gateway), a retry can be a pragmatic bridge. However, the retry should be logged as a "Soft Failure" and tracked in your flake budget, never treated as a total Green.

Why This Matters Now

The cost of compute is rising, and the complexity of distributed systems is making "test noise" more dangerous. According to [Scientific Direct], the impact of flakiness includes wasted resources, developer frustration, and the erosion of trust in automated gates. When developers stop trusting CI, they start shipping bugs.

Sources & Further Reading

According to [Talent500], Google’s battle against flakiness involves advanced mitigation strategies, yet 16% of their tests still exhibit non-determinism.
Research published by [Google] highlights that 84% of CI transitions are flakes, emphasizing the need for automated root cause location.
According to [Scientific Direct], test flakiness is a multi-dimensional problem spanning code, infrastructure, and execution environment.

Turning Flakes into Fixes with Operate

Treating every flaky test like a production incident is the right move for reliability, but it is incredibly labor-intensive for humans. This is where Operate changes the math.

Operate treats flaky CI as one of its proactive surfaces. Rather than letting a retry hide the signal, Operate opens its own cases on recurring flakes. It runs the same context-gathering, root-cause investigation, and verification pipeline it uses for production incidents. By the time an engineer looks at the flake, Operate has already identified the race condition or shared state issue and drafted a PR to fix it—treating the flake exactly like the miniature incident it is.