Stuck state: the failure mode behind both July 2 postmortems

TL;DR: Recent postmortems from Railway and CircleCI reveal that systems often capture a "bad state" during brief windows of network instability and hold it indefinitely after the underlying issue is resolved. This "stuck state" creates silent failures where health checks pass but performance is severely degraded, requiring explicit state re-assertion rather than simple recovery.

Engineering teams often operate under a comfortable hierarchy of failure: either a system is healthy, or it has crashed. We design our monitoring around these binary states, assuming that once a transient network blip or upstream outage clears, our systems will naturally return to a state of equilibrium.

However, the incident postmortem analysis of two major outages on July 2, 2026, tells a different story. In reports released by Railway and CircleCI, a more insidious pattern emerged. These were not cases of systems failing to recover, but rather systems recovering into a "wrong" state—a state that looked healthy enough to avoid paging but was broken enough to degrade the user experience.

This phenomenon, which we identify as "stuck state," occurs when a system captures a path, a route, or an address during a moment of maximum entropy and never checks if that information is still valid once the environment stabilizes.

Two outages, one failure mode

On July 2, 2026, both Railway and CircleCI experienced significant service disruptions. While the initial triggers differed, the core mechanism that prolonged the outages was identical: the accumulation of stale, incorrect state that outlived the original trigger.

In both instances, the infrastructure didn't just break; it "latched" onto the failure. This creates a dangerous gap in modern observability. When the root cause (the trigger) disappears, monitoring dashboards often return to green, even as the system's internal state remains corrupted. This leads to silent failure detection challenges where SRE teams are left wondering why latency is high despite all infrastructure being "up."

What Railway saw: connections that never re-check their route

The Railway US-East outage provides the most detailed look at how state can become pinned to a sub-optimal path. According to the Railway incident report, an upstream routing issue created a 20-minute gap in their network fabric. During this window, the system attempted to maintain connectivity by falling back to a slower management network.

The catastrophe wasn't the 20-minute blip; it was what happened after the fabric recovered. Railway found that their storage TCP connections, established during the instability, remained pinned to this management network. This path had roughly one-third the throughput of the standard fabric, leading to I/O wait times spiking to 58%. Because the connections were technically "established," no new connection was attempted that would have utilized the restored, high-speed path.

Simultaneously, Railway's private-network tunnels exhibited a similar "latching" behavior. Approximately 20,000 tunnels captured an incorrect peer address during the routing failure. Because many of these tunnels were idle, they never sent the specific packets required to trigger a re-discovery of the correct peer address. The state was stuck.

What CircleCI saw: threads that never time out

CircleCI’s July 2 outage followed a parallel trajectory but manifested in the application layer. According to the CircleCI incident report, a slow downstream data service began to lag, causing a backup in the API layer.

The primary issue here was a lack of aggressive timeouts. API threads began to wait for the slow service. As more threads entered a waiting state, they remained "stuck" even after the data service regained its typical performance. The thread pool became saturated with long-lived requests that had captured the "slowness" of the incident. Without a mechanism to force-clear these threads or a strict timeout to kill them, the system could not self-heal.

Key Takeaway: After the trigger clears, actively verify state instead of trusting green dashboards: long-lived connections, tunnels, and pools can hold a bad path silently.

Stuck state, defined: when nothing re-asserts correctness

We can define "stuck state" as any internal system configuration—be it a cached DNS entry, a pinned TCP socket, a latched peer address, or a saturated thread pool—that is updated during a failure but never re-validated upon recovery.

In a perfectly resilient system, state is ephemeral and frequently re-asserted. In a "stuck" system, state is persistent and only updated on failure. The irony of modern high-availability design is that our efforts to make connections long-lived and "robust" often provide the very mechanism for this failure mode. If a connection never drops, it never has a reason to re-resolve its destination or re-evaluate its route.

Why it does not page: degraded is not down

The most terrifying aspect of the July 2 incidents is that they were largely silent to traditional paging systems.

A "down" system is easy to catch: 500 errors spike, health checks fail, and the pager goes off. A system in a "stuck state," however, is merely "degraded."

• Railway’s case: The storage was available, just slow. The throughput was capped, but not zero.
• CircleCI’s case: The API was technically responding, but at a fraction of its capacity.

Most health checks are shallow. They check if a port is open or if a simple heartbeat returns. They rarely check if the connection is utilizing the optimal network path or if the thread pool is 98% exhausted by stale requests. This leads to the "zombie dashboard" effect: everything is green, but the users are feeling the pain.

A checklist: finding state that outlives its trigger

To defend against stuck state, SRE teams must look beyond the initial recovery. Use this checklist during and after an incident:

1. Connection Age: Do you have a distribution of TCP connection ages? If the "average age" is much higher than your recovery time, you may have pinned connections.
2. TTL Enforcement: Are you relying on indefinite cache or connection lifetimes?
3. Active Probing: Does your system periodically "test" the optimal path, or does it only move to a fallback?
4. Pool Cycling: Do you have a mechanism to recycle thread or connection pools after a period of high latency?
5. Path Verification: After a network event, can you verify that traffic has returned to the primary fabric rather than staying on management or fallback routes?

What to instrument: management network utilization, blackholed links, pool saturation

Standard monitoring often misses the metrics that reveal stuck state. To detect these failures, you need to instrument the "internal choices" your infrastructure makes:

• Management Over-utilization: Alert when traffic on your management or fallback network exceeds a baseline for more than 5 minutes. This usually indicates pinned connections.
• I/O Wait vs. Throughput: If I/O wait is high but throughput is low, your system is likely throttled by a bad path.
• Idle Tunnel State: Monitor the delta between "connected tunnels" and "functional tunnels."
• Saturation Metrics: Track the percentage of your thread pool occupied by requests older than your 99th percentile latency.

Takeaway: the trigger is upstream, the damage is yours

The July 2 postmortems serve as a sobering reminder: the recovery of the trigger is not the recovery of the system. Whether it is an upstream routing change at an ISP or a temporary slowdown in a third-party API, the "trigger" is often outside your control. However, the subsequent "stuck state" is entirely within your infrastructure.

Systems do not naturally gravitate toward health; they gravitate toward the most recent stable state they were able to achieve. If that state was achieved during a crisis, it is likely suboptimal.

How Operate mitigates stuck state

At Operate, we've built our platform to handle exactly this class of silent, degraded failure. Unlike traditional monitoring that waits for a hard threshold to be crossed, Operate’s proactive agents are designed to recognize the "shape" of a system in distress.

Operate monitors infra, databases, and logs simultaneously, opening cases on latency regressions that never trip a pager. When a system enters a stuck state, Operate doesn't just see a "green" health check; it notices the discrepancy between expected throughput and actual performance. Our platform uses a specialized verification agent—employing a separate model to check the initial diagnosis—to ensure that the "root cause" isn't just a cleared trigger, but a lingering state issue.

By automating the investigation of silent failures, Operate ensures that your team isn't left hunting for pinned connections or exhausted pools three hours after you thought the incident was over.

Sources & further reading

• According to the Railway Incident Report, storage connections remained pinned to a management network long after the primary fabric recovered.
• According to the CircleCI Post Incident Report, the lack of thread timeouts allowed a downstream slowdown to saturate API capacity.