← Blog · Engineering Leadership · August 11, 2026 · 5 min read · By Operate Editorial Team

Your Triage Agent Solved Everything Nobody Needed Help With

Triage automation is scored on total cases handled, which is dominated by cases nobody needed help with. Score it on the residual instead, and the picture changes.

Your Triage Agent Solved Everything Nobody Needed Help With

Your Triage Agent Solved Everything Nobody Needed Help With

TL;DR: Most AI triage bots appear successful because they automate the "low-hanging fruit" that engineers would have solved in minutes anyway. To truly measure value, teams must ignore total ticket counts and score automation solely on the "residual"—the complex, expensive incidents that actually drain engineering capacity.

The bot closed 40 tickets and nobody noticed

The pattern is becoming familiar to any team that has experimented with ai on call automation. A team builds a triage bot, connects it to Sentry or PagerDuty, and watches the dashboard. In the first few weeks, the numbers look spectacular: dozens of tickets summarized, a handful of redundant alerts closed, and a high volume of "active" triage.

However, as reported by 12gramsofcarbon.com, these setups often start great but "devolve into slop surprisingly quickly." The fundamental issue is that the "easy wins" captured by the bot are exactly the problems an engineer would have solved in seconds. When the easy cases are gone, what remains are the heisenbugs, the noise, and the architectural debt. The result? Teams that roll their own triage agents often find themselves switching them off within a month because they aren't actually reducing the cognitive load of the on-call rotation.

Two arguments, both about the wrong variable

The industry conversation around putting agents in production has largely split into two camps, but both are arguably focused on the wrong variable.

  1. Permissions and Blast Radius: This camp focuses on "trust tiers." According to Pradeep Kandepaneni, teams often abandon "Tier 0" (read-only) agents too early, fearing a lack of impact without write access. Even high-profile failures are litigated through this lens. Following a February 2026 Financial Times report on AWS outages, Amazon’s position was that the event was caused by user error and misconfigured access controls, not the AI itself.
  2. Capability: This camp argues that we are simply one model generation away from agents that can handle the "hard" tickets.

Neither of these variables predicts whether an engineering team will still be using their triage bot in ninety days. A read only ai agent production tool can be perfectly safe and highly capable, yet still be useless if it only solves for the "head" of the ticket distribution.

The denominator problem

The failure of most triage automation is a measurement failure. Total cases handled is a vanity metric.

The distribution of operational work is heavily skewed. The vast majority of tickets are cheap to resolve—missing environment variables, known log noise, or simple retries. A small tail of complex incidents, however, consumes the majority of engineering hours and causes the most burnout. Automating the cheap head of this distribution produces a large number on a dashboard but recovers near-zero actual capacity. If you score your triage bot on the total denominator, you are measuring success based on the work nobody needed help with.

Key Takeaway: If you score triage automation on total cases handled, your metric is dominated by low-value tasks that provide zero relief to your engineering team.

Measure the residual

To find the true value of an AI agent, you must change the denominator. You should only score automation on the residual. The residual is the set of cases that meet at least one of these criteria:

Score your ai on call automation on this set alone. A bot that closes 100 tickets but fails on the three that actually ruined someone’s sleep is a net-zero asset. An honest failure rate on the residual is a better indicator of progress than a 99% success rate on the trivial.

Why the residual is hard, and it is not the model's fault

The residual is where the signal is ambiguous. These are incidents where the root cause is three microservices away from the alert, or where the "obvious" explanation is a red herring.

This leads to a phenomenon known as correlated failure: an LLM that generates a wrong hypothesis for a complex incident will almost always rate its own hypothesis as sound. If the model lacks the ability to verify its own logic against raw evidence, it simply contributes to the noise of the residual rather than clearing it.

Generation is cheap, verification is the scarce good

The practical consequence for SRE leads is clear: read only ai agent production output is only useful if a human can act on it without redoing the work. If a senior engineer has to spend 15 minutes re-tracing the agent's steps to see if the "root cause" is hallucinated, the agent has failed.

Verification is the scarce good. According to recent engineering walkthroughs of CVE triage tools, the most effective setups use deterministic tests to decide outcomes, while the model is used only for enrichment. Even Meta’s internal root cause analysis platform, as detailed by Meta Engineering, focuses on post-processing results into reviewable tasks and pull requests rather than autonomous actions.

This is why Operate was built on the assumption that "read-only" is a permanent constraint, not a temporary one. To be useful on the residual, the hard part isn't the generation of a summary—it's the quality of the diagnosis. This requires a separate verification layer where a different model checks the primary agent's work, ensuring that correlated errors don't reach the human responder. The goal is an output that looks like a patch file with evidence attached, which an engineer can review and approve in seconds.

What to do on Monday

If you are currently evaluating or building a triage agent, stop looking at the total volume of tickets.

  1. Identify the Residual: Pull your incident data from the last quarter. Filter for tickets that took more than 30 minutes to resolve or that were escalated.
  2. Score the Agent: Run your current agent against that specific set of "expensive" tickets.
  3. Define a Kill Criterion: Decide now what success looks like on the residual. If the agent can't provide verifiable evidence for at least 20% of those cases by month three, it’s time to rethink the approach.

Sources & further reading

#ai agents#sre#on-call automation#incident response