504 Gateway Timeout Means a Timer Expired, Not a Slow Hop

A 504 gateway timeout is the status a gateway, reverse proxy or load balancer returns when it stopped waiting for an upstream server. In engineering terms, it is a statement about one component's timer, usually Nginx, a cloud load balancer or a CDN, and not a diagnosis of the application behind it.

We see teams lose hours on 504s by starting in the application logs, where the request often looks successful because it finished after the gateway had already given up.

In this post we cover what the status really tells you, how to draw the path, the timeout ladder and the three ways it breaks, what happens to the abandoned work, the order we read an unexpected 504 in, and the cases where the timer is not the problem.

What a 504 Gateway Timeout Tells You About the Path

The definition is narrow on purpose. RFC 9110 says the 504 status code "indicates that the server, while acting as a gateway or proxy, did not receive a timely response from an upstream server it needed to access in order to complete the request."

Two words do the work there. "Gateway or proxy" tells us which component wrote the response, and "timely" tells us it was measured against that component's own budget. The upstream may have been slow, stuck, unreachable or just slower than an unrealistic timer.

We keep its neighbour in mind, because the two are often confused. A 502 means the gateway got an invalid or unusable response, such as a refused or reset connection, while a 504 means it received nothing in time. A 502 points at a broken upstream, and a 504 points at a race between timers.

Draw Every Hop Before Changing Any Timeout

A request in a modern stack rarely goes straight from client to application. We draw the path first, because every hop on it runs its own clock.

• A CDN or edge proxy in front of everything.
• A web application firewall, sometimes built into the CDN.
• A cloud load balancer, such as an AWS Application Load Balancer.
• An ingress controller or reverse proxy, very often Nginx.
• A service mesh sidecar, if the cluster runs one.
• The application server and its HTTP client settings.
• The database, with its own statement and lock timeouts.

That list easily holds six to ten timers for a single request. Most teams have never written them down in one place, and the first 504 investigation is usually the first time anyone tries.

Three Ways the Timeout Ladder Breaks Behind a 504 Gateway Timeout

We call the ordered list of timers the timeout ladder. The table below holds the defaults we check first, each read from the vendor's own documentation.

Hop	Setting	Default	What it limits
AWS Application Load Balancer	Target connection timeout	10 seconds	Opening a connection to the target
AWS Application Load Balancer	Idle timeout	60 seconds	Waiting for the target to send data
Nginx	proxy_connect_timeout	60 seconds	Opening a connection to the upstream
Nginx	proxy_read_timeout	60 seconds	The gap between two reads from the upstream
Nginx	proxy_next_upstream_timeout	0, meaning no limit	Total time spent trying further upstream servers
PostgreSQL	statement_timeout	0, meaning disabled	How long one statement may run

The Nginx proxy module docs add a detail that changes how we read the 60 seconds. The read timeout "is set only between two successive read operations, not for the transmission of the whole response." A slow upstream that trickles bytes never trips it.

From those defaults we check one invariant for every adjacent pair. The outer hop's budget must exceed the inner hop's budget multiplied by the number of attempts the outer hop makes, plus connection time. When a pair breaks it, the 504 is waiting for a slow day.

1. Inversion, Where the Inner Hop Outlasts the Outer One

The application allows 90 seconds for a report, but the load balancer in front of it gives up at 60. The load balancer returns a 504 at second 60 while the application keeps working. The user sees a failure, and the application log later shows a success.

2. Equality, Where Two Timers Race

Nginx and the load balancer are both set to 60 seconds. Which one fires first depends on when each one started its clock, so the hop that emits the 504 changes from request to request. Logs from different hops stop agreeing, and the investigation goes in circles.

3. Unbounded Inner Timers

The PostgreSQL docs state that for statement_timeout "a value of zero (the default) disables the timeout." With no inner limit, a single slow query can hold its connection for as long as it likes, and every outer timer is guaranteed to lose the race. We treat any unbounded timer on the path as an inversion that has not fired yet.

The fix we prefer in all three cases is to lower the inner budget rather than raise the outer one. A failure that is fast, attributed and cheap beats one that is slow and anonymous.

What Happens to the Work the Gateway Abandoned

A timeout is not a cancellation. When an outer hop gives up, the inner hop usually has no idea the client has gone, so it keeps holding a worker, a database connection and its locks until the work finishes.

Retries make that worse, and some are built in. The Nginx default for proxy_next_upstream is error timeout, so a timeout while reading the response header passes the request to the next upstream server. The docs note that non-idempotent methods such as POST are not passed on once a request has been sent, and that retrying is only possible if nothing has reached the client yet.

So an idempotent request that times out can land on a second backend while the first one is still busy with it. On a path with several retrying layers, one slow dependency becomes a queue that never drains, and we cover how to measure that in retry storm diagnosis.

Reading an Unexpected 504 in the Order That Narrows It

During an incident we follow the same four steps, because each one shrinks the list of suspects before the next.

1. Time the failure. A 504 that arrives at almost exactly 10, 30 or 60 seconds matches a default in the ladder, and the number usually names the timer.
2. Identify the emitting hop from the response. The error page, the Server header and vendor trace headers such as X-Amzn-Trace-Id show which box wrote it.
3. Read that hop's logs first, not the application's. The application often recorded a success because it finished after the gateway gave up.
4. Only then look upstream for the slow part, such as a query, a lock or a dependency call.

Step four is where root cause analysis gets expensive, because the evidence sits in the load balancer logs, the database and the code at once. It helps to connect an investigator read-only to your logs and database so that cross-check does not depend on who happens to be on call.

The same logic applies one layer down in gRPC, where the error names the caller whose deadline expired, as we explain in context deadline exceeded.

When a 504 Gateway Timeout Is Not a Timeout Problem

We do not assume every 504 is a ladder problem. The ALB troubleshooting guide lists causes that no timer change will fix, such as a network ACL that blocks return traffic on ephemeral ports, or a target that sends a Content-Length header larger than the body it delivers.

A connection that never opens is a different failure as well. When the handshake itself stalls, the clue is in the connection timeout rather than the read timeout, and we treat it as a network problem first, as in TLS handshake timeout diagnosis.

And sometimes the work really is long. A report that legitimately takes three minutes should not be served synchronously through a 60-second load balancer, and raising every timer only hides that design choice. We move that work to a background job and let the client poll for the result.

Audit Your 504 Gateway Timeout Ladder This Week

We would list every hop, read the effective timeout at each one from the running configuration, check the invariant for each pair, and set a limit on every unbounded timer, starting with the database. Each inversion found that way is a 504 that never reaches a user.

A 504 gateway timeout is only confusing until the ladder is on paper. When one still needs explaining, Operate can help. It brings the evidence from logs, databases and code together through read-only adapters, and proposes any code fix as a patch file an engineer reviews and applies.