← Blog · Engineering Strategy · August 11, 2026 · 5 min read · By Operate Technical Editorial

Your Triage Agent Solved Everything Nobody Needed Help With

Triage automation is scored on total cases handled, which is dominated by cases nobody needed help with. Score it on the residual instead, and the picture changes.

Your Triage Agent Solved Everything Nobody Needed Help With

Your Triage Agent Solved Everything Nobody Needed Help With

TL;DR: Most AI triage bots appear successful because they automate high-volume, low-complexity tasks that humans would have resolved quickly anyway. To evaluate the true value of an AI agent, you must measure its performance on the "residual"—the difficult incidents that escalate or consume significant human time—where verification, not just generation, is the primary constraint.

The bot closed 40 tickets and nobody noticed

The pattern is becoming predictable. A team integrates background agents to watch every Sentry error or PagerDuty alert. For the first few weeks, the dashboards look exceptional. The agents summarize logs, link to relevant documentation, and close out dozens of low-level tickets.

However, as reported by engineering teams building these systems, the setup often starts great but "devolves into slop surprisingly quickly." According to [12gramsofcarbon], the easy wins are almost always problems an engineer would have solved in minutes regardless. What remains is the noise, the heisenbugs, and the high-stakes incidents. Consequently, many home-grown triage bots are quietly switched off within a month because they fail to move the needle on actual engineering capacity.

Two arguments, both about the wrong variable

When these tools fail to gain long-term adoption, the post-mortem usually focuses on one of two things:

  1. Permissions and Blast Radius: This camp focuses on "trust tiers." As [Pradeep Kandepaneni] argues, autonomy should be scoped to reversibility, starting at Tier 0 (read-only, human executes). Even the most high-profile agent failures are litigated this way. Following the February 2026 Financial Times report on the "Kiro" incident, Amazon’s position was that the outage was caused by user error and misconfigured access controls rather than the AI itself, according to [GeekWire].
  2. Model Capability: This camp blames the LLM. They argue that if we just had a larger context window or better reasoning, the triage bot would be reliable.

Both arguments miss the mark. Neither permissions nor model quality predict whether a tool is still in use after ninety days. The failure isn't that the agent is too dangerous or too "dumb"; it’s that it is being measured on the wrong denominator.

The denominator problem

If you score triage automation on total cases handled, you are using a vanity metric. Operational work follows a power-law distribution. The "head" of the distribution is a massive volume of trivial errors—temporary network blips or known deprecated warnings. The "tail" is the set of incidents that actually eat your team's night and weekend.

Automating the head of the curve produces a large "tickets closed" number, but it recovers near-zero human capacity. If an engineer takes 30 seconds to close a noisy alert, and the bot takes 5 seconds, you haven't actually saved the team. You've just automated the part of the job that wasn't a burden.

Key Takeaway: If you score triage automation on total cases handled, your success metric is dominated by the tasks that provide the least value to your engineering team.

Measure the residual

To understand if an AI agent is actually working, you must ignore the bulk of its activity and measure the residual. The residual is defined as the set of cases that meet any of these three criteria:

Score your triage automation on this set alone. An honest failure rate here is often 70-80% for basic agents, and that is where the real work begins. A "good" result isn't closing 100 tickets; it's providing the correct root cause for one incident that would have otherwise taken three hours to debug.

Why the residual is hard, and it is not the model's fault

The residual is difficult because it involves ambiguity. These are cases where the change that caused the failure occurred two systems away, or where the logs are misleading.

The primary problem here is correlated failure. If an LLM produces a plausible-sounding but incorrect root cause, it will usually rate its own hypothesis as highly sound if asked to double-check. Without a mechanism to ground the agent in hard evidence, the "Tier 0" read-only triage bot becomes a source of more work, not less, because the human must redo the entire investigation to verify if the bot is hallucinating.

Generation is cheap, verification is the scarce good

The practical consequence of the residual is that read-only output is only useful if a human can trust it without re-running the logs. According to [Charles Wayne], even in specialized fields like CVE triage, the model should only enrich the data while deterministic tests make the final decision. Similarly, [Meta's root cause analysis platform] post-processes results into reviewable tasks and pull requests rather than just narrating the problem.

This is the core design philosophy behind Operate. We operate on the assumption that read-only is a permanent constraint for many organizations. This forces the focus onto diagnosis quality. To avoid correlated errors, Operate uses a separate verification agent running a different model to audit the initial findings. The output is not just a chat message; it is a patch file with a linked evidence trail, shaped for a human to review and merge.

What to do on Monday

If you are currently evaluating or building an agentic ops tool, take these three steps:

  1. Pull last quarter's incidents: Filter specifically for the "residual"—anything that escalated or took over 30 minutes.
  2. Score the agent: Run your current automation against that set alone. Disregard every other "success."
  3. Set a kill criterion: Decide now what percentage of the residual the bot must handle correctly to justify its seat at the table. If it can't help when things are actually hard, it's just more noise.

Sources & further reading

#ai agents#sre#incident response#triage automation#llms in production