Retry Storm Diagnosis Starts With the Amplification Factor

A retry storm is a failure mode where clients retry a large share of failing requests and the extra traffic deepens the failure they are reacting to. Twitter's Finagle documentation puts it plainly in its glossary, describing retries that have "the effect of multiplying the volume of traffic sent to the unhealthy peers, exacerbating the problem."

We rarely find that retries started an outage. They usually amplify a fault that began somewhere else, and that distinction decides what to do at 3am.

In this post we cover the three states retries can be in, four ways to measure the amplification factor, why layered retries multiply, how to tell a storm from a traffic spike, a safe test that settles it, and when retries should be left alone.

Retries Are Reacting, Amplifying or Sustaining the Outage

When a dependency is overloaded and traffic to it is up, the useful question is not what a retry storm is. It is what happens if we cut retries right now. We place the system in one of three states first.

1. Reacting, where retries rescue a small share of requests from short blips and add little load.
2. Amplifying, where retries add enough load to make a struggling dependency fail more often.
3. Sustaining, where the original fault has cleared but queued retries keep the dependency saturated.

Cutting retries helps in the second and third states and hurts in the first. Telling them apart takes one number, measured per dependency rather than for the service as a whole.

Four Ways to Measure the Retry Storm Amplification Factor

We define the amplification factor as physical attempts sent to a dependency divided by the logical requests that caused them. A value near 1.0 means almost every request needed one attempt. The illustrative numbers below show how the factor separates a storm from a spike when raw traffic looks the same.

Signal	Baseline	Retry storm	Traffic spike
Logical user requests	100 per second	100 per second	1,000 per second
Physical dependency calls	105 per second	1,000 per second	1,050 per second
Amplification factor	1.05	10.0	1.05

For a threshold we borrow from Google. Its SRE book describes a per-client retry budget that only retries while retries stay below 10% of requests, which caps growth at about 1.1x. A factor well above 1.1 tells us retries are running past any sensible budget.

Most teams do not have a retry counter ready when the incident starts. These are the sources we use, in order of accuracy.

1. Group Trace Spans by Logical Request

We group spans by the request ID and count the calls to the suspect dependency under each parent. This is the most precise method, and it also shows which layer issued the extra attempts.

2. Read Proxy and Mesh Retry Counters

Proxies often retry outside application code, where nobody looks. Envoy, for example, exposes upstream_rq_retry for total retries and upstream_rq_retry_overflow for requests not retried because of circuit breaking or an exhausted retry budget, both listed in its cluster statistics.

3. Compare Gateway and Service Request Rates

We compare the inbound rate at the edge with the outbound rate from the calling service to the dependency. When the edge is flat and the outbound rate is climbing, the gap is amplification inside the system.

4. Watch Connection Pool Checkouts

If tracing is also degraded, the database or HTTP connection pool still counts checkouts. A surge in checkouts with no matching surge in user sessions points to a retry loop, and a pool that stays saturated after load drops is the sustaining state. Pool starvation has its own failure pattern, which we examined in the WorkOS pool exhaustion analysis.

Collecting these four views by hand during an incident is slow, because they live in traces, proxy metrics and database stats at once. This is the kind of cross-referencing that an AI agent that investigates with evidence is built for, working from read-only access while the on-call engineer decides what to change.

Layered Retries Multiply Instead of Adding Up

A retry policy that looks modest at each layer becomes large across a call chain. The Google SRE book chapter on addressing cascading failures gives the arithmetic. If the backend, frontend and JavaScript layers "all issue 3 retries (4 attempts), then a single user action may create 64 attempts (4^3) on the database."

The detail people miss is the difference between retries and attempts. Three retries means four attempts, so three layers with three retries each produce 64 attempts, not the 27 a quick multiplication of retries suggests.

The same chapter recommends retrying only at the layer immediately above the one that is rejecting requests. In practice we find the extra layers in places nobody configured on purpose, such as an ORM's default, a cloud SDK, a service mesh or a message consumer that redelivers. Messages that keep failing should leave the loop, which is the job of a dead letter queue.

Timeouts decide when each of those layers fires. When an outer hop gives up sooner than the hop behind it, a timeout can become a retry against a backend that is still busy with the first attempt, so a mismatched timeout ladder and a retry storm often arrive together. We cover finding that mismatch in the 504 gateway timeout ladder.

Telling a Retry Storm Apart From a Traffic Spike

Three failure shapes look identical on a raw traffic graph. We separate them with the signals below before touching any retry setting.

Signal	Retry storm	Traffic spike	Cache stampede
Inbound user traffic	Flat	Rising	Flat
Amplification factor	High and rising	Near 1.0	Near 1.0
What leads	Error rate rises before load	An external event or launch	A cache expiry or a deploy
Request mix	The same requests repeated	New requests from more users	Many distinct requests for one key

A cache stampede, often called a thundering herd, is the one most often mistaken for a storm. It is many different requests arriving at once because a shared cache entry expired, not the same request repeated, so retry limits do nothing for it.

A Scoped Test That Settles It Safely

When we suspect retries are sustaining an outage, we do not turn them off everywhere. We run a scoped relief test through a feature flag or dynamic configuration.

• Pick one non-critical caller or one route.
• Cap its retries at zero or one.
• Watch the dependency's load and success rate for a few minutes.

If success recovers and load drops, retries were sustaining the outage. If load drops but errors persist, retries were amplifying and the dependency is still broken. If nothing changes, the load comes from elsewhere, such as background workers, health checks or a real spike.

When Retries Are Innocent and Should Stay On

We leave retries alone when the amplification factor sits near 1.0. In that state they are rescuing requests from transient errors, and removing them lowers the success rate without reducing load in any meaningful way.

Retries are also pointless for deterministic errors, and they still add load. A 4xx response or a bug that fails the same way every time will not succeed on the second attempt, so the fix belongs in the client or the code rather than the retry policy.

Finally, if a circuit breaker or retry budget has already tripped, the damage is bounded. The SRE book's chapter on handling overload describes servers returning an "overloaded; don't retry" error so clients stop retrying, and when that mechanism is working we look for the original fault instead.

What the Postmortem Should Record About Retries

We want the postmortem to show the evidence, not a belief that retries were to blame. That means two curves on one chart, logical requests and physical attempts for the affected dependency, plus the amplification factor over the incident window.

It should also name the layer that produced the extra attempts and the setting that allowed it, such as a default in a client library or a mesh retry policy. Those two facts turn a vague action item like reviewing retries into a specific configuration change with an owner.

The search matters as much as the answer here, because the next storm will start in a different layer. We argue for keeping that record in postmortems that keep the search.

Measure Before You Switch Off Retries in a Retry Storm

We would measure the amplification factor per dependency, find the layer producing the extra attempts, and run a scoped test before changing any global setting. Root cause analysis then turns to the fault that started the failures, which retries only made louder.

A retry storm is a measurement problem before it is a configuration problem. When the original fault is still unclear, Operate can help find it. It brings the evidence from logs, databases and code together through read-only adapters, and proposes any code fix as a patch file an engineer reviews and applies.