AT&T's 1990 collapse is told as a missing break statement. The transferable lesson is that the defect was in recovery code, the self healing machinery was the vector, and 114 identical switches shared one bug.

The Bug Was in the Recovery Code: What AT&T's 1990 Collapse Says About Fleets That Heal Themselves

TL;DR The 1990 AT&T long-distance network collapse was caused by a defect residing exclusively in the recovery software of its 4ESS switches. While often framed as a simple coding error, the incident demonstrates that self-healing infrastructure can become the primary vector for failure propagation when identical instances share correlated recovery logic.

2:25pm, Bedminster, New Jersey

On January 15, 1990, the network operations center in Bedminster, New Jersey, watched as a 72-screen video array began to glow red. A malfunction was leapfrogging between switching centers across the United States. Standard procedures, designed to isolate faulty hardware, were failing to arrest the spread.

For the next nine hours, approximately 50 percent of all AT&T long-distance calls failed to connect. According to Dennis Burke’s account for Cal Poly, this resulted in an estimated $60 million in lost revenue from unconnected calls alone. The network only stabilized around 11:30pm, partly because the total call load finally fell below the threshold required to trigger the failure cycle.

The trigger was a correct action

To understand the collapse, one must look at the mechanism of the trigger. The failure began in a New York switch that correctly self-tested and found it was near its load limit. Following standard protocol, it performed a four-second maintenance reset and signaled to the rest of the network to stop sending it calls.

This was the system working as intended.

Upon recovery, the New York switch began draining its backlog and signaling its neighbors that it was back in service. A second switch, receiving this "back in service" message while simultaneously processing other signaling data, hit a race condition. It received two messages within a ten-millisecond window, which triggered a defect that overwrote internal communications data.

The second switch detected the data corruption and, in a bid to "self-heal," failed over to its backup processor. However, the backup processor hit the exact same condition. As that switch recovered and drained its own backlog, it sent the same sequence of messages to its neighbors, triggering the same failure in them. The self-healing mechanism had become the infection vector.

Why the story is usually told wrong

Most modern retellings focus on the "missing break statement" in the C code. While the pseudocode provided by Dennis Burke confirms a logic error in a switch statement, focusing on the language syntax misses the strategic lesson.

The C language is the least transferable part of this story. Three other factors are far more relevant to modern self healing infrastructure:

The defect lived exclusively in the recovery code.
The automated recovery mechanism was the propagation vector.
The fleet was perfectly homogeneous.

Key Takeaway: Redundancy across identical instances is not redundancy against a shared defect, and the code that runs only during failure is the code you have tested least.

Recovery code is the least exercised code you own

In 2026, every service has "ghost paths" that only execute during failure: leader election logic, cache rebuilds after a flush, replica promotion, and retry budgets. These are covered by unit tests, but they are almost never exercised by production-scale load.

The AT&T software update had been rigorously tested and had even operated through the high-volume Christmas peak without incident. It failed on January 15th because it required a specific, high-concurrency race condition during a reset—a scenario that only occurred when the recovery logic was actually needed.

Homogeneity is a single point of failure that no diagram shows

AT&T operated 114 identical 4ESS switches. In modern terms, this is equivalent to running the same container image across every replica, the same sidecar agent for automated remediation, and the same library versions pinned by a lockfile.

While this homogeneity simplifies deployments, it ensures that failure modes are correlated rather than independent. Architectural redundancy protects against hardware failure; it does nothing against a defect that every instance shares. To reduce this correlation, teams must rely on staged rollouts with significant "bake time" specifically for recovery paths, not just the request path.

The 2026 version of paranoid democracy

AT&T referred to its switch-monitoring system as a "paranoid democracy," where modules continuously checked each other for sanity. Modern engineering has built its own version:

Liveness probes that trigger restarts.
Readiness gates that shed load.
Circuit breakers that open under latency.
Autoscalers that add capacity based on signal.

Each of these is a control loop with the authority to act. The specific failure shape to watch for is a loop whose recovery action increases the very load that produced its trigger. If a restart adds cold-cache latency to a struggling cluster, the "healing" action becomes the "failure" vector.

Nine hours and 100 technicians

The most overlooked part of the 1990 collapse is the diagnostic clock. It took nine hours and 100 technicians working through the data to identify a one-line code fix. They were fighting a wall of 114 correlated alarms with no causal ordering. As the incident spread, the alarm count increased, making the problem look "bigger" without telling the operators where it actually started.

What to actually do on Monday

To prevent your automated systems from becoming a propagation vector, consider this checklist:

Inventory Control Loops: Identify every system that can act on production without a human (autoscalers, liveness probes, automated failover).
Analyze Signal Feedback: For each loop, ask: "Does the action taken (e.g., a restart) improve or degrade the signal that triggered it (e.g., latency)?"
Stress-Test Recovery: Identify recovery paths that have never run under 80% production load.
Order Your Alarms: Ensure your monitoring preserves "first occurrence" ordering so you can find the first switch to fail in a sea of red.

Sources & further reading:

According to Dennis Burke (Cal Poly), the January 1990 incident showed the possibility for all modules to "go crazy at once" despite redundant processors.
The timeline and impact figures are corroborated by Telephone World.
For the standard framing of the C bug, see Engineers Codex.

The lesson AT&T's own account draws is that self-healing software can inadvertently bring down healthy systems. This risk is why Operate is designed with a "read-only" investigative posture. Instead of granting an agent the authority to trigger potentially recursive recovery loops, Operate investigates across code, infrastructure, and logs to propose a specific patch. By drafting a PR for human review, it addresses the nine-hour diagnostic problem without becoming the next propagation vector.