Railway and CircleCI both went down on July 2, 2026. The interesting failures came after recovery: connections and workflows that silently kept bad state.

When Recovery Doesn’t Recover: Railway, CircleCI, and Stuck State

TL;DR The outages at Railway and CircleCI on July 2, 2026, revealed a critical failure class known as "stuck state," where systems capture bad routing or connection paths during brief instability and maintain them indefinitely after infrastructure recovers. While primary network carriers restored service within minutes, these platforms suffered hours of silent degradation because internal state—such as mesh tunnels and database connections—never re-asserted their correct configuration, leaving "green" dashboards masking a crippled system.

July 2, Twice: A Tale of Two Timelines

On July 2, 2026, two major engineering organizations faced disruptions that appeared, on the surface, to be standard infrastructure hiccups. However, the subsequent postmortems revealed a deeper, more systemic issue. While the initial triggers were different—a carrier-level network degradation for Railway and a combination of maintenance and high-volume deletions for CircleCI—the tail of these incidents was remarkably similar. Both systems entered a state where the outage technically ended, but the failure persisted.

Railway’s Timeline: The 20-Minute Gap and the Two-Hour Tail

According to the Railway incident report, the trouble started when a tier-1 network carrier, Arelion, experienced significant packet loss in the US-East region. This caused a 20-minute gap where default routes were unreachable. However, the true incident began when the network "recovered."

As the primary network path failed, internal routing protocols attempted to find any available path. In many cases, systems latched onto a management network—an out-of-band path designed for low-bandwidth administrative traffic, not for high-throughput production database I/O. When the primary network came back online, approximately 20,000 mesh tunnels remained anchored to this slow management path. The system was "healthy" according to basic pings, but production traffic was being blackholed or throttled by an order of magnitude.

CircleCI’s Timeline: The Unending Workflow

Simultaneously, CircleCI was dealing with its own version of persistent degradation. According to the CircleCI postmortem, a confluence of scheduled maintenance and an unusual spike in customer-driven data deletions led to a resource exhaustion event.

The critical failure here wasn't just the initial spike; it was the lack of connection timeouts. API workers became tethered to slow or unresponsive backend processes. Because these connections didn't have strict TTLs (Time to Live) or aggressive timeout policies during the period of instability, they remained "stuck." Even after the maintenance window closed and the deletion spikes subsided, circular dependencies kept workers in a state of thread exhaustion. In some cases, workflows were still appearing as "Running" a week later because the state transition to "Failed" or "Timed Out" was never triggered by the system's recovery logic.

The Shared Failure Class: Stuck State

These two incidents highlight a phenomenon we call Stuck State. This is a failure class where a system captures a suboptimal or "bad" configuration path during a period of instability and fails to re-evaluate that path once conditions return to normal. It is the architectural equivalent of a "stuck pixel" on a screen—the rest of the display is updating, but that one point remains fixed in a defunct state.

Why Stale Connections After Failover Happen

In modern distributed systems, we optimize for connection persistence. Re-establishing a TLS handshake or a mesh tunnel is expensive in terms of CPU and latency. Therefore, we design systems to hold onto connections as long as possible.

The danger is that these connections don't automatically re-check routes. If a TCP session is successfully established over a degraded path (like Railway’s management network), the operating system and the application layer have no natural incentive to drop that connection and seek a better one. This leads to stale connections after failover. The infrastructure has "failed over" to a backup, but it never "fails back" to the primary because the backup is technically functioning—just poorly.

Tunnels and Peers: The "Sticky" Problem

In the case of mesh networking (Wireguard, Tailscale, or proprietary overlays), the problem is exacerbated. Tunnels often rely on endpoint discovery. If a peer discovery event happens during a network split, the tunnel might associate a peer with a temporary, low-performance IP.

Once that association is written to the local state table, it stays there. Unless there is a "re-assertion" mechanism—something that forces the system to ask, "Is this still the best way to reach my peer?"—the system will continue to use the bad path. This is a silent degradation that evades standard connectivity checks because the peer is technically reachable.

Threads and Timeouts

CircleCI’s failure demonstrates stuck state at the application level. When threads don't time out, they become a permanent "leak" of system capacity. Recovery restored the network and the database availability, but it didn't restore the availability of the thread pool. The system’s state (the consumed threads) was fixed in the "occupied" position, regardless of the health of the underlying resources.

Key Takeaway: Stuck state occurs when a system’s recovery logic assumes that restoring infrastructure health automatically restores application state, ignoring the "bad paths" captured during the window of instability.

Why Monitoring Missed It

The most frustrating aspect of these July 2 outages for the engineers involved was the "Green Dashboard" problem. By all standard metrics, the systems appeared to be recovering.

Routing Tables Looked Correct: At the VPC and BGP levels, the routes were advertised correctly. The "stuckness" happened inside the individual application tunnels and connection pools, which are often opaque to infrastructure-level monitoring.
The Cluster Was "Healthy": Standard Kubernetes liveness probes or Consul health checks often look for "Can I talk to this service?" If the answer is "Yes," even if the latency is 5000ms over a management network, the node is marked healthy.
Pinned Throughput: Throughput didn't drop to zero; it dropped to exactly the capacity of the management network or the size of the un-exhausted thread pool. To a high-level monitor, this can look like a normal dip in traffic rather than a systemic failure.

This highlights the danger of silent degradation. Traditional monitoring alerts on binary states (Up/Down). Stuck state requires alerting on path and utilization—knowing not just that a packet got there, but which wire it took to get there.

Hunting Stuck State in Your Own Stack

To prevent your system from outliving its next outage, you must move from "event-driven recovery" to "continuous state re-assertion."

Re-assertion as a Design Principle

Your system should never trust its long-term state. Implement "Anti-Entropy" routines that periodically tear down and rebuild connections, even if they appear healthy.

Max Connection Age: Set a MaxConnectionAge on load balancers and database pools. This forces clients to re-resolve DNS and re-establish paths.
Tunnel Re-keying: Configure mesh networks to periodically re-verify peer endpoints, forcing a discovery lookup that would move traffic off a management network and back onto the backbone.

Fail-Closed Fallbacks

If a system must fail over to a management or "emergency" path, it should do so with a loud alarm and a TTL. Ideally, these paths should be "fail-closed" for production traffic. It is often better for a service to be hard-down (triggering an immediate SRE response) than to be "slightly up" on a path that can only handle 1% of the load, causing a stuck state postmortem later.

Alerting on Path and Latency Deviations

Instead of just monitoring HTTP 200 rates, monitor the identity of the underlying paths.

Source IP Tracking: Alert if production traffic is being sourced from management IP ranges.
Latency Outliers: Use Heatmaps, not averages. A "stuck state" often manifests as 10% of your connections having 100x the latency of the others. An average will hide this; a heatmap will show a distinct, horrifying line of "stuck" traffic.

What an Automatic Investigation Would Have Seen

If an AI-driven or automated investigation tool had been observing the July 2 incidents, the "stuck state" would have been surfaced in minutes rather than hours.

In Railway's case, an automated traces of the kernel would have shown an I/O wait spike of 58% or higher on specific nodes, despite the network being "up." By cross-referencing the socket metadata, it would have identified that the active connections were suddenly sourced from administrative addresses rather than production interfaces.

In CircleCI's case, a look at the thread dumps during the "recovery" phase would have shown hundreds of threads stuck in an established state with no data moving, pointing directly to the lack of application-level timeouts. The evidence was there; it was simply buried under the assumption that "if the network is back, the system is back."

Why This Matters

This failure class is becoming more common as our systems become more "self-healing." Automation is excellent at finding any path to keep a system alive during a crisis. However, the same "intelligence" that finds a path through a management network during an ISP failure is the same logic that keeps you trapped there once the ISP recovers.

As environments move toward highly abstracted mesh networks and complex sidecar architectures, the distance between "network state" and "application state" grows. Engineers can no longer assume that a green light from the cloud provider means their internal state is healthy.

Sources & Further Reading

According to the Railway Engineering Blog, the primary issue was 20,000 tunnels failing to re-assert state after a carrier outage.
According to CircleCI's incident report, thread exhaustion persisted long after the triggering events due to a lack of timeouts.
According to Network World, the Arelion disruption on July 2 was a global event that tested the failover capabilities of thousands of organizations.

At Operate, we see this "green dashboard, degraded system" pattern frequently. Our self-hosted AI SRE platform is designed to look past surface-level health checks, investigating the "why" behind the latency. By verifying the actual evidence of a fix—such as ensuring connections have moved back to production paths—Operate helps engineering teams catch silent degradations before they become multi-hour postmortems. This provides the auditability and control needed to ensure that when your infrastructure recovers, your system actually does, too.