Context Deadline Exceeded Names the Caller, Not the Slow Hop

Context deadline exceeded is the error Go returns when a context is cancelled because its deadline passed, and it surfaces across Kubernetes, gRPC services, Helm, Prometheus and anything else built on Go. We read it as a receipt that someone gave up, not as a diagnosis of who was slow.

The reflex fix is to raise the timeout in the service that logged the error. That usually moves the error somewhere else or hides a dependency that is getting worse.

In this post we cover what the error tells you, where the expired deadline was set, how the deadline budget shrinks across hops, four hypotheses in the order that narrows fastest, and when raising the timeout is the right call.

What Context Deadline Exceeded Tells You and What It Hides

The Go context package documentation defines it precisely. DeadlineExceeded is the error returned by Context.Err when the context is cancelled because its deadline passed, and its text is the string you see in logs.

That is all the string records. It names the component whose context expired, which is the component that chose to stop waiting. We treat it as the observer of the problem rather than its source.

The message is silent on the three things we need most during an incident:

• Which downstream hop was slow, or never answered at all.
• Which layer set the deadline, and how long it was.
• Which of two stories is true, a dependency that degraded or a budget that was always too small.

Every useful step in the investigation answers one of those three questions, so we start by writing them at the top of the incident notes.

Where the Deadline That Expired Was Actually Set

Deadlines are designed to travel. When a context is derived from a parent, Go adjusts the new deadline to be no later than the parent's, so a child can never wait longer than its caller allows. That means the deadline that fired is often set one or more hops away from the code that logged it.

We start by mapping the surface where the error appeared to the setting that most likely controls it. These are the defaults we check first, each taken from the tool's own documentation.

Surface	Setting that controls the deadline	Documented default
Go HTTP client	http.Client.Timeout or a context passed to the request	Zero, which means no timeout
gRPC client	A deadline set on the call	None, so a call can wait indefinitely
Kubernetes probe	timeoutSeconds	1 second
Helm upgrade	--timeout	5m0s per Kubernetes operation
Prometheus scrape	scrape_timeout	10 seconds

The Helm value comes from the helm upgrade reference and the Prometheus value from its configuration reference. When the error comes from a probe, the probe design matters more than the number, which we cover in our liveness probe decision guide.

How the Deadline Budget Shrinks at Every Hop

The arithmetic operators rarely write down is the deadline budget. A five-second deadline set at the edge is not five seconds at the third hop, because every hop spends part of it before calling the next one.

The gRPC deadlines guide is direct about the starting point. "By default, gRPC does not set a deadline which means it is possible for a client to end up waiting for a response effectively forever." Once a deadline is set, though, it follows the request.

Consider a chain we see often, with the budget at each step:

1. The edge gateway sets a five-second deadline and calls service A.
2. Service A spends 800 ms on its own work, then calls service B with about 4.2 seconds left.
3. Service B calls service C, gets an error, and retries twice with a 300 ms backoff between attempts.
4. Each retry starts with less time than the attempt before it, and the last one runs out partway through.

Service B logs context deadline exceeded. The useful question is not why B was slow, because B may have done almost nothing. It is where the budget went before B ever started.

• Retries Inside an Inherited Deadline

Retries under an inherited deadline are often doomed before they start. Each attempt gets the remainder of the same budget, so a retry policy written for a generous deadline can turn a single slow call into three failed ones and more load downstream. When the retries themselves start driving load, we check the amplification first, as we describe in measuring the retry amplification factor.

• Propagation Across Services and Languages

The gRPC guide notes that propagating a deadline from an incoming call to an outgoing one is enabled by default in Java and Go and must be switched on explicitly in some other languages, such as C++. A mixed-language chain can therefore propagate the budget through some hops and drop it at others. We confirm propagation hop by hop rather than assuming it.

• The One Number Worth Logging

Measuring how long a call took is not enough. We log the remaining budget on entry to every handler, read from ctx.Deadline(), which reports the deadline, or that none is set. With that single field, a log line showing a hop that entered with 50 ms left explains the error by itself.

Four Hypotheses for Context Deadline Exceeded, in Order

We test these four in the same order every time, because each one is cheaper to rule out than the next.

1. The Budget Was Always Too Tight

The operation's tail latency has sat just under the deadline for weeks, and a small shift in traffic pushed it over. We compare the deadline with the p99 of the call over the last month, and our p99 latency field guide covers how to read that tail.

2. A Dependency Genuinely Degraded

A query that usually takes 10 ms now takes 200 ms and eats into the budget of every caller above it. We look for the downstream span or log line whose own duration changed, not the callers that inherited the delay, and this is where root cause analysis usually ends.

3. The Call Is Hung, Not Slow

A connection waiting on a stalled TCP session, an exhausted pool or a DNS lookup that never returns looks identical to a slow call in the error. We check whether the dependency returned any bytes at all before the deadline. None at all points to waiting rather than working.

4. Clocks Disagree Between Machines

Where a deadline is compared as an absolute time on two machines, clock drift can make it fire early. Temporal's troubleshooting guide lists checking system clocks first for its deadline-exceeded errors. The gRPC guide explains that gRPC converts a deadline into a timeout with the elapsed time already deducted, which shields plain gRPC calls from skew, so we check clocks last for those.

When several hops each have their own timer, the same evidence can show one of them set in the wrong order, which is the subject of our timeout ladder audit.

Finding the Slow Hop With or Without Tracing

The procedure differs depending on what telemetry we have, but the goal is the same. We want the hop that was actively spending time just before the cancellation arrived.

• With Distributed Tracing

A cancelled parent cancels its children, so a trace of a failed request often shows a wall of red where every span reports an error. We ignore the cancelled status and look for the span with the longest self time just before the cancellation. That span is the first suspect.

• Without Tracing

Logs can carry the same answer if every outbound call records three fields:

• The operation name and the target it called.
• The start time and the total duration.
• The remaining deadline budget on entry.

If service A entered with 500 ms remaining and its call to B took 501 ms, the log shows the bottleneck directly. Because the evidence sits in several places at once, it helps to connect an investigator read-only to your logs and database so the timeline is assembled before anyone starts guessing.

When Raising the Timeout Is the Correct Fix

Raising the timeout is the right answer in one situation we see regularly. The deadline was set arbitrarily, often a default or a round number from years ago, and the operation is healthy and simply needs longer.

Large exports, Helm releases with long hooks and long model calls fit that pattern. Their latency is stable and within expectations, and the deadline was never tied to anything a user experiences.

In those cases we set the deadline from a service level objective rather than a guess, and we raise it at the layer that owns the request, not at every hop below it. Raising it only in the hop that logged the error leaves the upstream deadline in charge and changes nothing.

Follow the Budget Before You Raise a Context Deadline Exceeded Timeout

We would start with where the request entered and what deadline it carried, then find the first hop that logged the error and the dependency it was calling. Only once one hop clearly spent the budget does a timeout change make sense.

Handled that way, context deadline exceeded becomes a short trace of where the time went rather than a timeout tweak. That is the kind of evidence trail Operate assembles. It brings the evidence from your logs, databases and code together through read-only adapters, and proposes any code fix as a patch file an engineer reviews and applies.