A Flaky Tests Runbook That Treats Each Flake as an Incident

Flaky tests are tests that pass and fail against the same code, with no change to the test or the system under test. We treat each one as an incident in miniature, because a test suite that people have learned to distrust stops protecting anything.

The common response is to rerun until green and move on, and eventually to delete the test. That loop removes the noise without ever learning why the test was unreliable, and sometimes the reason was a real bug.

In this post we cover why flaky tests deserve an incident response, how to detect them from data, a six-step runbook from ticket to verified fix, the team rules that keep it working, and when a flaky test is really a product bug.

Why Flaky Tests Deserve an Incident Response

Google published some of the clearest numbers on the problem in its testing blog post on flaky tests at Google and how it mitigates them. It reported that about 1.5% of all test runs produced a flaky result, and that almost 16% of its tests had some level of flakiness.

The figure we find most telling is about trust. Google found that about 84% of the transitions from pass to fail in its continuous integration system involved a flaky test, so most red builds were not new regressions at all.

That is how teams learn to ignore failures. Once engineers expect a red build to be noise, a real regression arrives looking exactly like another flake, and it gets rerun instead of investigated. We think that cost justifies treating each flake with the same discipline as a small production incident.

Detect Flaky Tests From Data Instead of Complaints

Complaints identify the flakes that annoy the loudest team, not the ones that fail most. We detect flakiness from CI history instead.

The cleanest signal is a test that fails and then passes on the same commit. Any CI system that records reruns can produce a list of these flips per test, and a flake rate per test follows directly, as the share of runs on unchanged code that failed.

The pytest documentation on flaky tests offers a useful frame for what the data means. "Broadly speaking, a flaky test indicates that the test relies on some system state that is not being appropriately controlled." It also notes that flakes appearing only when the suite runs in parallel can indicate a dependence on test ordering, and lists plugins that randomise test order to expose state problems.

We set a simple entry threshold, such as any test that flips on the same commit twice in a week, and open a ticket automatically when a test crosses it.

The Flaky Tests Runbook in Six Steps

Each step below has an output, so the ticket shows where the test is in its life. We run them in order for every flake that crosses the threshold.

1. Open a Ticket With a Named Owner

A flake without an owner stays flaky. We route the ticket to whoever owns the test file, and GitHub's documentation on code owners describes the standard way to record that, a CODEOWNERS file that defines the individuals or teams responsible for code in a repository.

The ticket records the test name, the first and latest failing runs, the flake rate and the commit range.

2. Capture the Failing Run's Evidence

Evidence disappears fast once a test has been rerun. We keep the full log of a failing run, the runner and its resource usage, the test order and random seed, timestamps and any service or container logs from the same window.

This is the same evidence an incident responder would want, and it is where most flaky test investigations stall. Some teams script the collection, others use an AI agent that investigates with evidence to pull logs and recent changes into the ticket before anyone starts guessing.

3. Quarantine the Test With Exit Criteria

Quarantine takes the test off the merge path while it keeps running and reporting. Google describes the trade-off honestly. Quarantining "removes the test from the critical path and files a bug for developers to reduce the flakiness", which helps developers but could easily mask a real problem.

We avoid silent skips for that reason. A quarantined test still runs on every build, its results go to a dashboard, and its ticket carries a release date or a review date so quarantine cannot become permanent deletion by another name.

4. Classify the Root Cause

Classification is the root cause analysis step, and it turns a mystery into a short list of likely fixes. A study by Luo, Hariri, Eloussi and Marinov at the University of Illinois, which looked at 201 commits that likely fix flaky tests in 51 open-source projects, classified the most common root causes and the strategies developers used to fix them. We use a working taxonomy along the same lines.

Root cause	Typical signal	Diagnostic move	Fix pattern
Asynchronous wait or race	Fails more on slow or busy runners	Add a debug-only delay and see if the flake disappears	Wait on an explicit condition, never a fixed sleep
Leaked state between tests	Passes alone, fails in the full suite	Run it right after the suspected polluting test	Reset databases, caches and globals in teardown
Test order dependency	Fails under parallel or random order	Replay the exact order and seed from the failing run	Make each test create its own state
Network or third-party dependency	Connection refused or timeout errors in the log	Check the failing run's outbound calls	Stub external services or use contract tests
Time and date	Fails at midnight, month end or across time zones	Set the clock to the failing run's timestamp	Inject or freeze the clock
Randomness and unordered results	Assertion on order or generated values	Loop the test hundreds of times locally	Fix seeds or sort before asserting

5. Fix the Cause, Not the Symptom

A longer timeout or an automatic retry makes the symptom rarer without removing it. We accept those only as temporary containment, recorded on the ticket, and look for the fix the classification points to.

When the cause sits in the application rather than the test, the ticket becomes a bug report for the product team, with the evidence from step 2 attached.

6. Verify With Enough Consecutive Green Runs

One green run proves almost nothing about a test that failed 2% of the time. The arithmetic is simple. If the flake were still present at rate p, the chance of n consecutive passes is (1 minus p) to the power n, so we keep running until that chance falls below 1%.

• For a flake rate of 5%, about 90 consecutive green runs.
• For a flake rate of 2%, about 228 consecutive green runs.
• For a flake rate of 1%, about 459 consecutive green runs.

We run that loop in the same CI environment where the test flaked, ideally across several runners at once, because a laptop loop can pass forever while the CI runner keeps failing.

Team Rules That Keep the Flaky Test Runbook Working

The runbook fails without a few standing rules. We keep three.

• The number of quarantined tests is visible on a team dashboard and reviewed regularly, so quarantine cannot grow quietly.
• Every quarantined test has an owner and a date, and an expired date triggers a decision to fix, rewrite or delete.
• Automatic retries are a policy decision with limits, never a default that hides every failure.

Retry policy and measurement deserve their own treatment. We cover how to retry without letting a regression through in retrying flaky tests safely, and how to put a number on the damage in measuring the impact of flaky tests.

When a Flaky Test Is Really a Product Bug

The runbook assumes the test is the problem, and often it is not. A test that fails only under load or on slower runners may be exposing a real race condition that users will eventually hit in production.

We read a flake as a report about the whole system first. Before anyone rewrites the test to pass, the owner should rule out that the application itself behaves non-deterministically, because a test made to pass around a real race hides the bug for good.

The opposite case exists too. Some tests duplicate coverage elsewhere or test behaviour the product no longer has, and for those, deleting the test after a short review is the honest fix. We also distinguish flaky tests from pipelines that pass without testing anything, a different failure we describe in CI/CD pipeline failures that stay green.

Run Your Next Flaky Test Through the Runbook

We would start with the single flakiest test in the suite. Give it an owner and a ticket, keep the evidence from its next failure, quarantine it with a date, classify the cause, fix it, and release it only after the green-run count says the flake is gone.

Doing that once usually shows the team how much of its flaky tests problem is really unexamined evidence. Examining evidence is what Operate is built for, reading logs, databases and repositories through read-only adapters and proposing any code fix as a patch file an engineer reviews and applies.