The flaky test runbook: detect, quarantine, diagnose, verify

TL;DR: Flaky tests are non-deterministic test results that undermine CI/CD reliability and developer trust. To fix them, teams must treat each flake as a production incident—implementing a structured runbook that includes automated detection, strict quarantine with exit criteria, and statistical verification of the fix.

The most common advice for managing flaky tests on community forums like Reddit is often a cynical "just delete them." While tempting, this represents a failure of engineering discipline. When a test suite becomes non-deterministic, it ceases to be a safety net and becomes a source of ambient noise that masks real regressions.

For high-performing engineering teams, flaky tests are not a minor annoyance; they are silent incidents. This guide provides an operational runbook for moving beyond the "delete-retry-ignore" loop and toward a rigorous system of flaky test management.

Why flaky tests persist: the delete-retry-ignore loop

Most engineering organizations handle test flakiness through an informal, three-stage cycle that leads to technical debt. First, a test fails unexpectedly. The developer, suspecting a "glitch," hits the retry button. If it passes, the failure is ignored. Eventually, the test flakes so often that it blocks a critical PR, at which point a frustrated developer comments it out or deletes it entirely.

This loop persists because we treat flakiness as a property of the test code rather than a failure of the system. According to Datadog, a flaky test is a test that provides different results (pass or fail) across multiple runs against the same code version. When we allow these results to exist without investigation, we train developers to ignore CI failures—the "normalization of deviance."

Treat it as an incident class, not an annoyance

The primary shift in perspective required to solve flaky tests is to treat them as an incident class. In production, we don’t ignore a 500 error just because it only happens to 1% of users. We investigate the root cause.

Treating a flake as an incident means:

1. Assigning Ownership: Someone must be responsible for the "resolution" of the flake.
2. Evidence Collection: Saving the logs, traces, and state of the failing run immediately.
3. Containment: Quarantining the test so it no longer impacts the main CI pipeline.
4. RCA (Root Cause Analysis): Identifying why the non-determinism exists.

Key Takeaway: Run flaky tests through the same discipline as production incidents: measure flake rate, triage by root cause taxonomy, quarantine with exit criteria, and verify fixes statistically instead of hoping.

Detection: flake rate, not anecdotes

You cannot fix what you cannot measure. Relying on developer complaints to identify flaky tests is inefficient and biased toward the most vocal teams. Instead, implement automated detection based on the "flake rate."

A flake rate is the percentage of times a test fails in an environment where no code changes have occurred. Modern CI observability tools can track this by looking for "flip-flops"—instances where a test fails and subsequently passes on the same commit hash.

What causes flaky tests? (The Detection Threshold)

Before a test is officially labeled as "flaky," it should cross a threshold. For example, if a test fails 2% of the time across 100 runs without a code change, it is a candidate for the incident runbook. Identifying what causes flaky tests at scale requires looking for patterns: Does it only fail at night? Does it only fail on certain CI runners? Does it only fail when the full suite is run in parallel?

Triage: is it the test, the suite, or the app?

When a test flakes, triage is the first operational step. A common mistake is assuming the test code itself is buggy. In reality, flakiness often stems from the environment or state leakage.

• Leaked State: An "innocent" test fails because a previous test didn't clean up its database records or global variables. This is common in flaky unit tests where shared mocks aren't reset.
• The App: The application code has a genuine race condition that only manifests under the specific resource constraints of a CI runner.
• The Suite: Resource exhaustion (CPU/Memory) on the CI agent causes timeouts that don't happen locally.

The five root causes: diagnostic moves and fix patterns

To efficiently fix flaky tests, use a taxonomy of the five most common root causes. Each has a specific diagnostic move to prove the theory.

1. Race Conditions (Concurrency)

This is the "main cause of flaky tests" in modern asynchronous applications. A test expects an element to be present, but the background process hasn't finished yet.

• Diagnostic Move: Add a manual sleep (for debugging only) to see if the flake disappears.
• The Fix: Replace sleeps with deterministic polling or "wait-for" assertions.

2. Leaked State (Pollution)

Tests that pass in isolation but fail when run as part of a large suite.

• Diagnostic Move: Run the failing test immediately after the suspected "polluter" test.
• The Fix: Implement strict tearDown hooks that clear databases, caches, and global state. According to Jason Swett at CodeWithJason, reproduction is the hardest part of this phase; running tests in a random order can help surface these leaks.

3. Network and Third-Party Dependencies

Tests that call external APIs or rely on external DNS resolution.

• Diagnostic Move: Check CI logs for ECONNREFUSED or ETIMEDOUT.
• The Fix: Use Contract Testing or extensive mocking/stubbing for external services. Never let your CI's green light depend on a third party's uptime.

4. Time and Date Dependence

Tests that fail on the first of the month, during leap years, or across timezone boundaries.

• Diagnostic Move: Manually set the system clock on the CI runner to the time of the reported failure.
• The Fix: Use time-freezing libraries (like Timecop or Freezed) to mock the system clock.

5. Randomness and Unordered Collections

Tests that rely on UUID generation or assume that a database query will always return results in the same order without an ORDER BY clause.

• Diagnostic Move: Run the test 100 times in a loop locally.
• The Fix: Force deterministic seeds for random number generators or add explicit sorting to assertions.

Quarantine with exit criteria, never silent skips

Once identified and triaged, a flaky test should be moved to a quarantine suite. This is a critical part of flaky test management. A quarantined test still runs (so you can collect data), but its failure does not "break the build" or block merges.

The Golden Rules of Quarantine:

1. Visibility: The number of quarantined tests should be a KPI for the engineering org.
2. No Silent Skips: Do not use it.skip or @Ignore. Use a specific attribute that allows the CI to run the test and report results to a dashboard without failing the pipeline.
3. Exit Criteria: A test cannot stay in quarantine forever. It must have a "Return to Service" (RTS) date or a Jira ticket attached.

Verification: how many green runs is enough?

One of the greatest pitfalls in resolving flaky test cases is the "single green run" fallacy. If a test fails 1% of the time, a single pass proves nothing.

According to guidelines from the Mill-build engineering blog, verifying a fix for a low-frequency flake requires statistical significance. If a test has a 1-in-50 failure rate (2%), you need approximately 300-500 consecutive green runs in the CI environment to be 99% confident the flake is gone.

Runbook Action: Use a "stress-test" job that runs the suspected test in a tight loop across multiple parallel nodes before allowing it out of quarantine.

The org policy layer: retries, merge gates, ownership

While the technical fixes are vital, flaky tests are ultimately a policy problem.

• The Cost of Retries: Auto-retrying tests at the CI level (e.g., in GitHub Actions or CircleCI) is a common "band-aid." However, this hides the problem and increases your "CI Minutes" spend. If a suite has a 5% flake rate and you retry every failure, you are effectively paying 5-10% more for your CI for the privilege of ignoring your bugs.
• Merge Gates: Should a flaky test block a merge? Ideally, no—provided it is in quarantine. If it isn't in quarantine yet, it should be treated as a blocker until the "incident" is triaged.
• Ownership: Shared code makes for orphaned tests. Every test file should be mapped to an owner (a team or person) via CODEOWNERS.

Why this matters: The cost of 5 percent

When your flaky test rate hits 5%, developer productivity doesn't just drop by 5%—it craters. Developers begin to "context switch" while waiting for retries, pipelines take twice as long to clear, and the "Cry Wolf" effect leads to actual regressions slipping into production because everyone assumed the failure was "just another flake."

Common Pitfalls

• The "Flaky" Label as a Catch-all: Don't label every hard-to-debug failure as a flake. Sometimes it's just a bug that's hard to reproduce.
• Indefinite Quarantine: A "quarantine" that never ends is just a "delete" with extra steps. Set an expiration.
• Mocking Everything: Over-mocking to avoid flakiness can lead to tests that pass but don't actually test the integration, leading to a false sense of security.

Sources & further reading

• According to CodeWithJason, the key to fixing flakes is consistent local reproduction using specific environment variables.
• According to Mill-build.org, an automated system to "detect and disable" is superior to manual maintenance.
• According to Datadog, CI visibility tools are essential for identifying the "flip-flop" patterns of non-deterministic tests.
• According to the Reddit r/programming community, the lack of ownership is why most flaky test initiatives fail.

Operationalizing flaky test resolution

Managing flaky tests at scale is an operational burden that most teams are under-equipped to handle. It requires constant monitoring of CI outputs, deep-dive investigations into logs, and the generation of verified patches.

This is exactly where Operate changes the math. As a self-hosted AI SRE platform, Operate monitors your production and CI environments as a proactive watcher. When a test flakes, Operate doesn’t just notify you; it treats it as a case. It spins up a specialized pipeline to conduct the RCA—investigating leaked state, race conditions, or network dependencies by examining the evidence.

Instead of a developer spending hours on the "delete-retry-ignore" loop, Operate drafts the fix as a PR with the evidence attached. For teams that value control and privacy, keeping this diagnostic power self-hosted ensures that your code and CI logs never leave your infrastructure, while your test suite stays deterministic and trusted.