The Recorder Ran Out Before Anyone Knew There Was an Incident: What the FAA's 25-Hour Rule Says About Your Retention Window

TL;DR: The FAA recently extended cockpit voice recorder requirements from 2 hours to 25 because critical incident evidence was consistently being overwritten before investigators could secure it. Most engineering teams make the same mistake, setting a log retention policy based on storage costs rather than the time it actually takes an organization to recognize a silent failure has occurred.

San Francisco, 7 July 2017

On a clear night at San Francisco International Airport, an Air Canada A320 was cleared to land on runway 28R. Instead, the crew lined up on taxiway C, where four fully loaded wide-body aircraft were waiting for takeoff. The A320 descended to just 100 feet—clearing the tail of one aircraft by mere feet—before performing a go-around. It was nearly the greatest disaster in aviation history.

According to the Federal Register (91 FR 4447), the NTSB investigation into case DCA17IA148 faced a critical wall: a 46-hour notification delay. By the time Air Canada officials realized the severity of the event and notified investigators, the Cockpit Voice Recorder (CVR) had already overwritten the audio from the approach. The investigation lacked direct evidence of the flight crew's decision-making process during those critical seconds because the data simply no longer existed.

The window was sized against the wrong thing

For decades, the standard for CVRs was a rolling two-hour window. This was a design choice based on the assumption that an "incident" is something known immediately upon landing. If a plane crashes, the recorder stops. If a plane has a "near miss," the crew is supposed to pull the circuit breaker to preserve the data.

The FAA has now admitted this logic was flawed. Two hours was sized against flight duration, not against organizational recognition time. Preservation depended on a human knowing in the moment that the event mattered. According to NTSB report ASR1804, at least 14 events between 2003 and 2018 resulted in overwritten CVR data. A January 2024 NTSB letter to the DOT noted at least 14 more such instances since 2018. The system worked perfectly, but the window was too small to capture the reality of how organizations process information.

What the FAA did about it

On February 2, 2026, the FAA published the final rule (91 FR 4447) mandating a shift from two hours to twenty-five hours for new aircraft. They didn’t just make a minor tweak; they increased the retention window by 1,250%.

Crucially, the FAA did not tell investigators to work faster or demand that pilots become better at recognizing trauma in real-time. They bought the window. The incremental unit cost was estimated at approximately $5,209 per aircraft in 2024 dollars—a small price to pay for the integrity of an investigation. They recognized that human recognition delay is a variable you cannot optimize through policy; you can only solve it with storage.

Software has the same arrangement and has not noticed

Most software organizations operate under the "Two-Hour CVR" mindset without realizing it. We set our log retention policy based on the observability bill rather than evidence grounds.

• Trace sampling is often set at one percent to save on ingest fees, meaning 99% of your "near misses" are never recorded.
• Log retention is capped at seven or fourteen days because "that’s enough for a sprint," ignoring that complex bugs often take weeks to surface.
• Ephemeral infrastructure like Kubernetes pods or serverless functions clear /proc state and in-memory ring buffers the moment they are restarted, acting like a pilot who clears the black box immediately after a near-collision.

Key Takeaway: Your log retention window should be sized against your organizational recognition delay—the time it takes to realize a failure happened—not the duration of the failure itself.

Your recognition delay is longer than you think

When we talk about incident response, we focus on the "PagerDuty to Resolution" time. But the FAA rule highlights a different metric: the time between the event occurring and the organization knowing it occurred.

In software, this delay is often days or weeks. Consider silent failures: an integration quietly dropping 5% of records, a reconciliation gap that only appears during end-of-month reporting, or a customer reporting a "weird UI glitch" eleven days after it first appeared. If your log retention policy is seven days, you are investigating an incident with an empty recorder.

The mitigation is often the deletion

In aviation, investigators preserve the scene. In software, our first instinct is to destroy it. We have runbooks that start with "Restart the service" or "Roll back the deployment."

While this mitigates the immediate impact, it often acts as a delete button for evidence. Restarting a pod releases file descriptors and clears the process view. Rolling back replaces the very binary that produced the error. We lack the "custody discipline" found in aviation because we prioritize uptime over understanding—often at the cost of fixing the same bug three times.

What to do instead of asking people to be faster

You cannot force your customers to report bugs faster, but you can change how you preserve the evidence they eventually report.

1. Measure Recognition Delay: Look at your last ten "High" or "Medium" severity incidents. Calculate the time from the first timestamp of the error to the time the incident was actually opened. If that median is 10 days and your retention is 7, your log retention policy is failing you.
2. Tail-based Sampling: Instead of a flat 1% sample, use tail-based sampling to keep 100% of traces that result in a 5xx error or exceed a latency threshold.
3. Capture-Before-Mitigate: Add a step to your incident response: trigger a heap dump or a snapshot of /proc before the restart command is issued.
4. Tiered Retention: Keep a narrow, high-value slice of logs (like audit logs or error-level events) for 30+ days while discarding high-volume debug logs earlier.

The uncomfortable version of the question

If a serious problem started in your system today and nobody noticed for two weeks, could you still find out what happened?

If the answer is no, you haven't made a technical decision; you've made a financial one. You have decided that the cost of storage is higher than the value of knowing why your system failed. Like the FAA before 2026, you are relying on luck and human speed to compensate for a window that is simply too small.

Root cause analysis is only as good as the trail that survived. This is why Operate is designed to read across code, databases, logs, and infrastructure rather than relying on a single, fragile source of truth. Because Operate is self-hosted and read-only, it can inspect the entire environment without destroying the evidence in the process. Its verification agent ensures that a proposed fix is backed by actual evidence—but even the best AI SRE cannot investigate what has already been overwritten. Don't let your recorder run out before you even know you're in trouble.

Sources & further reading

• According to the Federal Register (91 FR 4447), the FAA finalized the 25-hour CVR rule to prevent the loss of critical investigative data.
• According to NTSB report ASR1804, multiple investigations were significantly hampered by the 2-hour overwrite cycle.
• According to Aerotime, the new rule represents a significant shift in aviation safety standards, moving away from reliance on manual crew intervention.
• Further details on the Air Canada SFO incident can be found in NTSB Case DCA17IA148.