Which Flaky Test Causes Point at Bugs in the System

Flaky test causes are usually filed under noise, which is why teams answer them with a retry button. We think that habit throws away information. A test that fails once in twenty runs has found a path through the system that does not behave the same way twice, and sometimes that path also exists in production.

It would be wrong to say every flake is a product bug. Many are bugs in the test, and the useful skill is telling the two apart quickly.

This post lists the common causes with the evidence we have for how often each occurs, shows which of them point at the system, gives a way to sort a flake into test or product, and says when a flake really is only a test problem.

Flaky Test Causes Mostly Come From Waiting, Threads and Shared State

We lean on the best-known classification, which comes from the University of Illinois. In An Empirical Analysis of Flaky Tests, Luo, Hariri, Eloussi and Marinov studied commits that fixed flaky tests in open-source projects, and classified 161 of the 201 they inspected.

Cause category	Commits (of 161)	Share	What it means
Async wait	74	45%	The test uses a result before an asynchronous call has finished
Concurrency	32	20%	Threads interact badly through data races, atomicity violations or deadlocks
Test order dependency	19	12%	The outcome depends on which tests ran before, through shared state
All other categories	36	23%	Resource leaks, network, time, IO, randomness and similar causes

The study also found that most flaky tests, 78%, were flaky from the first time they were written. That matters for prevention. It says the cheapest place to catch a flake is the first week of a test's life, before anyone has grown used to it.

Which Flaky Test Causes Point at Bugs in the System

The same paper tells us where to look. For concurrency flakes, it reports that 10 of 32 cases, about 30%, were caused by non-determinism in the code under test and showed up as an intermittent test failure. It also notes that missing synchronization can cause bugs in the code under test even when no test ever flakes. We would sort the common causes with that in mind.

1. Async Waits That Hide Missing Synchronization

The test sleeps for two seconds and hopes a service has started. Most of the time that is a test bug, since the fix is to wait for a signal instead of a duration. It becomes a system question when the product itself offers no way to know it is ready, because real clients have the same problem.

2. Concurrency Flakes With a Production Twin

A data race or deadlock that appears under test load can appear under production load too. These are the flakes we would triage first, because the test is doing what a test is for, which is finding a defect before customers do.

3. Order Dependence Through Shared State

When test B passes only after test A, they share state that nothing cleans up. The pytest documentation frames the general case well. It says a flaky test indicates that the test "relies on some system state that is not being appropriately controlled." In a service, uncontrolled shared state is also the shape of a cross-request leak, so we check if the product shares that state between users.

4. Resource Leaks and Environment Dependence

Leaked connections, file handles and ports make a later test fail for an unrelated reason. A suite that leaks is often a service that leaks, and the failing test is simply the first to hit the ceiling.

Flake cause	Usually a test bug when	Points at the system when
Async wait	A fixed sleep stands in for a signal	The product exposes no readiness signal
Concurrency	The test shares a mutable fixture across threads	The race is inside the code under test
Order dependency	Fixtures are not reset between tests	The product shares state between users or requests
Resource leak	The test forgets to close what it opens	The code under test leaks the same resource

A Failed Run Holds Evidence That a Retry Destroys

An automatic retry answers the wrong question. It asks if the test can pass, when the useful question is why it failed once. The first failed run holds the timing, state and environment that exposed the problem, and a green second run overwrites that story.

Google's testing team makes the same point in its post on where flaky tests come from. It argues that if a test flakily fails, it should be considered a bug in the same way as a test that fails consistently. It also reports that larger tests are significantly more flaky than smaller ones, which suggests that part of the fix is shrinking what each test touches.

There is also a statistical reason to read the first failure. A test that fails one run in twenty fails about once a day in a team that runs a hundred builds, so the pattern is easy to dismiss and easy to see once the failures are collected in one place. We keep the failing runs together and look for what they share, such as a time of day, a runner type or a neighbouring test.

We would keep retries, but log each one as a soft failure and keep the artefacts of the first failed run. Our guide to retrying flaky tests without letting a real regression through covers the configuration, and our flaky test incident runbook covers handling one flake as an incident.

How to Sort a Flake Into Test Bug or System Bug

Sorting matters because the two outcomes need different owners. A test bug belongs to the team that wrote the test and is fixed in an afternoon. A system bug belongs to the team that owns the code, goes into the same queue as any defect, and deserves a fix that the original test can then verify.

We use five checks, in this order, and stop when one of them gives a clear answer.

1. Run the test alone, then in the full suite, then in reverse order. A change in outcome points at shared state.
2. Run it many times with a race detector or thread sanitiser where the language has one. A report names the data race directly.
3. Replace fixed sleeps with waits on a signal. If the flake disappears, the test was guessing about timing.
4. Run it under CPU and memory pressure. A flake that appears only under load often reflects a timing assumption in the product.
5. Check the logs of the failed run for the same error in production. A match turns a test problem into an incident.

The last check is the one that justifies the whole exercise. Searching production logs and code history for the failure signature is slow when done by hand, and it is the part where we would use dedicated AI agents that investigate problems and cite their evidence, while a person decides what the flake means.

For the cost side of the same problem, see our notes on the impact of flaky tests and how to measure it.

When a Flaky Test Really Is Only a Test Problem

Treating every flake as a product incident would waste the team's time, and we would not do it. The same study attributes only 10 of its 32 concurrency flakes to the code under test, and it warns that non-determinism in the code "may or may not be a bug", since correct code can have several valid behaviours that a test wrongly narrows to one. Fixed sleeps are also a test problem in most cases.

Tests that depend on a third-party sandbox are another exception. If the sandbox is unreliable, a retry with a recorded soft failure is a reasonable bridge, as long as the retry budget is visible and someone owns shrinking it.

Ownership is the other limit. A flake in a shared integration suite may have no clear owner, and the temptation is to retry it forever. We would assign every quarantined test an owner and an expiry date, so that a soft failure cannot become permanent background noise.

The rule we apply is simple. If the failure cannot occur outside the test harness, fix the test and move on. If a user could plausibly trigger the same sequence, treat it as a defect and file it as one.

Treat the First Failed Run as Evidence

Flaky test causes are mostly ordinary, which is why the exceptions matter. Keep the artefacts of the first failure, sort each flake with the five checks and give concurrency and shared-state flakes the quickest look.

We would begin with the three flakiest tests in your suite and sort them this week. When you want agents to gather the evidence from logs and code history while your engineers keep the decision, Operate runs dedicated AI agents that investigate production problems with cited evidence and propose fixes for your team to review.