Connection Reset by Peer Is a Decision, Not a Failure: Proving Which Component Sent the RST

TL;DR: A connection reset by peer error (ECONNRESET) indicates your kernel received a TCP RST packet for an established connection, immediately discarding the socket. This is an intentional act by a specific component—such as a load balancer, firewall, or service mesh sidecar—rather than a random network failure, and can be attributed by analyzing timing signatures and platform-level metrics.

What the error actually records

In production logs, this error appears as connection reset by peer, surfaced as ECONNRESET (errno 104) on Linux, errno 54 on macOS, or WSAECONNRESET (10054) on Windows. It is critical to correct the record on two common misconceptions:

1. It is not an HTTP status code. While search engines often surface queries for "HTTP code 104," 104 is a POSIX errno. There is no such thing as an HTTP 104 status.
2. It is not the same as "Connection Refused." A connection refused (ECONNREFUSED) occurs when a RST is received in response to your initial SYN packet because no listener exists on that port. ECONNRESET happens only after a connection is established.

When this error occurs, your kernel receives a TCP segment with the RST (Reset) bit set. It immediately tears down the connection, flushes the buffers, and notifies your application.

Why the reset destroys its own evidence

In a graceful shutdown, both ends exchange FIN packets, allowing in-flight data to be processed and log lines to be written. A RST is an abortive signal; it is a single segment that provides no acknowledgement of intent.

The primary difficulty in debugging connection reset by peer is that the component sending the RST often does so because it has already lost the state for the connection or is acting at a layer below the application. Consequently, the "other side" frequently has no log entry for the event. If a load balancer resets an idle connection, your backend application may never even know a connection existed. This is why "checking the other service's logs" is often a dead end.

Reset, broken pipe, refused, timeout: A four-way distinction

The six components entitled to send a RST

1. The Peer Application

The remote application may explicitly call abort() or set SO_LINGER with a timeout of zero. Most commonly, this happens if the application closes a socket while there is still unread data in its receive buffer.

• Signature: Fires at the moment of request completion.
• Evidence: Correlates exactly with application-level processing time.

2. The Peer Host's Kernel

If a host receives a packet for a TCP 4-tuple (source IP/port, dest IP/port) it doesn't recognize, it sends a RST. This happens during process crashes, container restarts, or when a client reuses a pooled connection that the server host has already forgotten.

• Signature: Fires on the first write after an idle gap.
• Evidence: ss or netstat on the peer host shows no matching ESTABLISHED socket.

3. Proxy or Load Balancer Idle Timeout

Managed services like AWS ALB, HAProxy, or Nginx have idle timeouts. If no data flows for $X$ seconds, they unilaterally send a RST to both the client and the server to reclaim resources.

• Signature: A fixed, "round" interval (e.g., exactly 60 or 300 seconds) with no traffic immediately prior.
• Evidence: Matches configured values like keepalive_timeout or alb.idle_timeout.

4. Stateful Firewalls and Conntrack

Network Address Translation (NAT) and firewalls maintain a "conntrack" table. If the table is full, or if a connection stays idle longer than the conntrack timeout, the firewall evicts the entry. Subsequent packets for that connection are treated as invalid and met with a RST.

• Signature: High nf_conntrack_count on the node.
• Evidence: In Kubernetes, check node-level dmesg for "nf_conntrack: table full, dropping packet."

5. The TLS Layer

During the TLS handshake, a mismatch in SNI, ALPN, or an expired certificate can trigger a RST before any application data is sent. According to autocomplete data, this is increasingly common with LLM providers (OpenAI, Ollama) and is often misreported as a transport error.

• Signature: Lands before the first byte of the HTTP response.
• Evidence: Nginx logs this as while SSL handshaking to upstream.

6. Service Mesh Sidecars (Envoy/Istio)

In a mesh, the sidecar manages the connection lifecycle. If a pod is being terminated, the sidecar (Envoy) might shut down before the application container finishes its last few requests, leading to resets at the edge of a deployment.

• Signature: Errors cluster tightly around rollouts or node drains.
• Evidence: Envoy access_log shows response flags like UC (upstream connection termination).

Attribution when you can capture packets

If you can run tcpdump, do not just look for the RST. Compare the IP TTL (Time to Live) and IP ID fields of the RST against successful packets from the peer. If the TTL of the RST is different from the TTL of the data packets, the RST was injected by a middlebox (firewall or proxy) rather than the peer host.

Attribution when you cannot capture packets

In 2026, most production environments use managed services (AWS ALB, API Gateway, Cloudflare) where packet capture is impossible. In these cases, use the Evidence Exclusion method:

• Check the Timing Histogram: If 90% of resets happen at the 60.0s mark, it is a load balancer idle timeout.
• Reproduction: Manually hold a connection open on a canary instance for just longer than the suspected timeout. If it resets, you have found the component.
• Cloud Metrics: Look for TargetConnectionErrorCount in AWS CloudWatch or nf_conntrack_max reached in your Kubernetes node metrics.

The Decision Table

Why this matters

The request path has grown significantly. A single call now traverses client pools, cloud load balancers, Ingress controllers, mesh sidecars, and kube-proxy NAT before reaching a managed backend. According to Devrim Ozcay, blaming the network is usually a waste of time because the network is merely the messenger for a configuration mismatch between these components.

Common pitfalls

• Raising timeouts: If a proxy is timing out at 60s, raising it to 120s often just delays the reset. The correct fix is making the client-side connection pool idle window shorter than the proxy's.
• Blind retries: Retrying a connection reset by peer on a non-idempotent operation (like a POST for a payment) is dangerous. If the reset happened after the server processed the logic but before the ACK, a retry causes a double charge.
• Ignoring Conntrack: Engineers often look at pod CPU/RAM while the underlying node's conntrack table is saturated, causing "silent" resets that look like application crashes.

Sources & further reading

• According to Stack Overflow, the error is fatal and signifies an abortive close.
• According to OneUptime, idle timeouts and stateful firewalls are the most frequent culprits in cloud environments.
• According to Samba.org, a RST is distinct from a FIN because it discards all data in the pipeline.

Key Takeaway: A TCP reset is not a failure of the network, but a deliberate decision by a specific component that can be identified by its timing signature and platform-layer metrics.

Identifying these patterns manually is the core of modern SRE work. At Operate, our AI SRE platform automates this by correlating connection pool metrics against conntrack saturation and proxy logs to find the evidence for a reset. By bucketing reset rates by their likely originator, Operate helps teams move past "network blip" excuses and provides the specific configuration PRs needed to align timeouts across the stack.