Automated Root Cause Analysis Tools Judged by Ten Questions

Automated root cause analysis is software that investigates an incident on its own, pulling signals from across the stack and proposing the most likely cause with evidence. Newer products in the category use large language models for that reasoning. We think the label now covers tools that differ enormously in what they read and what they are allowed to do.

That makes buying one harder than it looks. Much of what ranks for the term is a product page, and every product page promises faster answers. What the pages rarely show is how a tool behaves when the answer is not in the logs, or when it is wrong.

In this post we cover the layers a tool has to reach, the ten questions we would take into any demo, how to run a fair trial on incidents you have already solved, and when buying a tool is not yet worth it.

An Automated RCA Tool Has to Reach Five Layers

Production incidents rarely stay in one layer. A checkout error can trace back to a slow query, which got slower after a schema migration, which passed in staging because the seed data was small. A tool that only reads logs and metrics will point confidently at the symptom.

We check coverage against five layers before anything else, because a gap here caps everything that follows.

Layer	What the tool should read	The question it answers
Logs and traces	Error spikes, the first deviation from normal, cross-service traces	Where did the failure surface
Database	Slow queries, query plans, lock waits, recent migrations	Did the data layer change under the code
Infrastructure and config	Deploys, scaling events, config and feature flag changes	What changed in the last hour
CI/CD	The last good build, flaky tests, what shipped and when	Which release carried the change
Code	The diff in the incident window and who owns it	Which line actually caused it

A tool that covers only the first row is an observability product with a summary on top. That can still be useful, but it is a different purchase. We explain how the remaining layers fit together in which parts of the SRE job an AI SRE actually does.

Ten Questions That Separate Automated Root Cause Analysis Tools

We would ask every vendor these questions and insist on a live answer rather than a slide. They are ordered so the cheapest disqualifiers come first.

1. Which Layers Does It Read in One Investigation

Ask the vendor to name every data source a single investigation touches, then check the five layers above. "We integrate with your stack" is not an answer. We want to see the database and the code diff in the same run as the logs.

2. Does Every Hypothesis Come With Evidence

A good output reads like a careful engineer's incident note. It lists log lines with timestamps, the query plan, the deploy and the commit, and you can click through to the raw data. "The root cause is likely a database issue" with nothing behind it is a guess.

3. Does It Show What It Ruled Out

Negative evidence is what turns an answer into an audit trail. We want the leads the tool checked and dismissed, with the reason. This is also the main defense against a trap the Google SRE book lists among troubleshooting pitfalls, "Hunting down spurious correlations that are actually coincidences or are correlated with shared causes."

4. Is the Confidence Score Calibrated

A confidence number only helps if it matches reality. If a tool says 80% and is right 40% of the time, the score is decoration. Neural networks are not calibrated by default, and Guo and colleagues found that "modern neural networks, unlike those from a decade ago, are poorly calibrated." We ask for a calibration curve on real incidents, and ignore the score if the vendor has none.

5. Does a Second Model Check the Answer

A second model reviewing the first model's evidence catches some confident mistakes, and disagreement is a useful signal to route a case to a human, though we would not oversell it. Research on correlated errors in large language models found that models agree 60% of the time when both are wrong on one leaderboard dataset, even across different providers. So a second check reduces risk without removing the need for evidence.

6. What Credentials Does It Hold in Production

The right answer is read-only, scoped and audited. Anything that can write to the database, the cluster or the repository changes the risk of the whole purchase. OWASP traces this class of problem to excessive functionality, permissions and autonomy, and we go through the read-only setup layer by layer in investigating production without write access.

7. What Form Does the Fix Take

We prefer a patch file that goes through your normal review and CI, because everything downstream of a diff already works. Your code review, your rollback and your git history stay the audit log. An agent that applies changes itself needs a much stronger answer to question six.

8. Where Does It Run and Where Does Data Go

Ask where the agent runs, where its credentials live, and which model provider receives log lines and rows. These are three separate boundaries, and we compared the deployment shapes vendors sell in self-hosted vs cloud AI agents.

9. Can Non-Engineers Ask It Questions Safely

Support and product teams ask "is this the outage or something else" all day. A read-only tool that answers from Slack or Teams with evidence moves that load off engineers without adding risk. If the tool can write, opening it to the whole company is a much harder decision.

10. What Happens When It Is Wrong

Every tool is wrong sometimes, and Microsoft's RCACopilot paper reports RCA accuracy up to 0.766, a strong research result that still leaves room for misses. We want to know what a wrong answer costs. If the honest answer involves undoing a change the agent made, go back to question six.

How to Run a Fair Trial on Incidents You Already Solved

Demos are built on incidents the vendor chose. We learn more from a trial on our own past incidents, where we already know the answer and can grade the tool honestly.

1. Pick five to ten resolved incidents from the last year, with a mix of database, deploy and infrastructure causes.
2. Write down the confirmed root cause and the evidence that proved it before the trial starts.
3. Give the tool only the access it would have in production, read-only, and the same time window.
4. Score each run on correct cause, quality of evidence, what it ruled out, and time to a useful answer.
5. Note every case where it was confidently wrong, since those matter more than the average.

Two habits make the results trustworthy. We hide the postmortems from anyone setting up the trial, and we include at least one incident where the cause sat in code rather than infrastructure.

If you want to judge the output format before connecting anything, we publish root cause analysis run on public GitHub issues. Reading a few of those tells you quickly if a tool's evidence would satisfy your own reviewers.

Investigation Tools and Remediation Agents Are Different Purchases

Many products blur two jobs, finding the cause and changing production. We evaluate them separately, because they carry very different risks and need different approvals.

Question	Investigation tool	Remediation agent
Access it needs	Read-only across the stack	Write access to the systems it changes
Cost of a wrong answer	A wasted lead and some engineer time	A second incident on top of the first
Who approves the purchase	Engineering, with a security review	Engineering, security and often leadership
Output	Evidence and a proposed patch	A change already applied

We would buy the investigation half first and treat any automated action as a narrow, separate decision. A few well-understood steps, such as restarting a worker with a known leak, can be worth automating with a hard limit on blast radius.

When Automated Root Cause Analysis Is Not Worth Buying Yet

We like these tools, but there are situations where we would wait.

• Your Telemetry Has Gaps

A tool can only reason over what you collect. If logs are sampled heavily, traces are missing, or nobody records deploys, the tool will produce confident answers from partial evidence. Fixing collection first usually pays off more.

• You Have Very Few Incidents

A small team with one or two incidents a quarter may not recover the setup and review time. In that case a clear runbook and a habit of writing postmortems give most of the value. We looked at how one large company justified the cost at scale in Meta's automated investigations, and the numbers there do not transfer to a small team.

We also check that someone owns the tool's output. Without an owner, evidence-backed answers still go unread, and the investment buys nothing.

Bring the Ten Questions to Your Next Demo

We would print the ten questions, run a trial on incidents you already solved, and score every tool the same way. Weight evidence, calibration and credentials above speed claims.

Judged that way, automated root cause analysis becomes a purchase you can defend to security and to the engineers on call. We built Operate to pass that test. It runs inside your own infrastructure with read-only adapters, a separate agent on a different model checks each proposed root cause against the evidence, and any code fix arrives as a patch file an engineer reviews and applies.