Skip to content
    Back to blog
    Incident Response

    How to Run Root Cause Analysis in Software Without Stopping at the First Why

    Written by:OperateOperate TeamUpdated 5 min read

    Run software root cause analysis from evidence: build a timeline, test explanations, choose a method and validate corrective work.

    A single 5 whys chain that stops at one node becomes a fault tree whose AND gate needs three conditions.

    Root cause analysis in software is a process for explaining a failure from evidence and identifying changes that reduce the risk of recurrence. Start with the incident timeline, affected requests and relevant system changes. Test competing explanations before choosing a method such as 5 whys or a fault tree.

    Finding one plausible cause is not the same as establishing how an incident happened. A useful analysis also records the contributing conditions, gaps in the evidence and how corrective work will be checked.

    Restore service and preserve evidence

    During an active incident, service recovery takes priority. Keep the response coordinated so evidence collection does not obstruct mitigation or overload an already struggling system.

    Preserve available logs, traces, deployment records, configuration changes and incident notes for a relevant window. Record the source, time zone and collection time. Restrict access to evidence that contains customer information or secrets. An export copied to an incident document can create a second sensitive dataset.

    Choose the window from the observed symptoms and retention limits. A fixed extra 30 minutes will not fit every incident. If evidence is unavailable, record the gap rather than treating the absence of a record as proof that an event did not happen.

    Build one timeline from observable events

    Separate these timestamps:

    • The earliest supported customer impact
    • The first signal or alert
    • The start of investigation and mitigation
    • The change applied to restore service
    • The observation confirming recovery

    Use machine timestamps where available and record clock uncertainty. A deployment record and a log entry may use different clocks. The apparent order can be misleading if their time difference is smaller than that uncertainty.

    Keep service recovery distinct from completion of the cause investigation. A rollback can restore checkout while the mechanism behind the failure remains unresolved.

    Write explanations that can be tested

    Consider an illustrative payment incident. Errors rose after a release. One explanation is that a retry change increased concurrent requests. Another is that the payment provider slowed down independently.

    The first predicts a change in retry behavior and overlapping attempts. The second predicts increased upstream latency that is not explained by the release alone. Compare logs, request identifiers, configuration and code. Ask what evidence would contradict each explanation.

    A correlation is a lead. The latest deployment is not automatically the cause, and a successful rollback may change several conditions at once. State what the recovery demonstrates and what it does not.

    Google's effective troubleshooting chapter describes a hypothesis-driven approach. Use the same discipline in the incident record: retain rejected explanations and the observations that ruled them out.

    Choose the method after understanding the evidence

    Method Useful when Limitation
    5 whys Following a particular failure mechanism into contributing conditions A single chain can omit parallel conditions
    Fishbone Organizing candidate causes across code, data, configuration and dependencies A diagram lists hypotheses; it does not prove them
    Fault tree Describing combinations of conditions needed for a failure The model is only as complete as its evidence and assumptions
    Change analysis Comparing a known working state with a failing state A visible change can distract from latent conditions or external failures

    There is no required number of whys. Branch when the evidence supports more than one contributing condition. Avoid ending with “human error” when the system allowed a routine mistake to become a customer-facing failure.

    Connect each conclusion to evidence

    For every material finding, record the observation, source and inference. Distinguish a demonstrated mechanism from a plausible one. If a controlled reproduction is appropriate, use a suitable test environment and representative inputs. Do not reproduce a destructive condition against production just to make the explanation more persuasive.

    For code changes, inspect the actual repository revision and test output. A proposed patch, a passing isolated example and a passing repository suite are different forms of evidence. Missing dependencies or failed tests belong in the report.

    Turn findings into owned work

    Corrective work can prevent a trigger, limit its impact, improve detection or make recovery easier. Give each action an owner and a review date. Prefer a concrete validation step over an instruction to be more careful.

    For example, an illustrative retry incident might lead to a reviewed retry policy, an idempotency test and a check that abandoned requests do not keep consuming connections. Those are proposals to assess against the actual system, not a universal prescription for payment failures.

    Google's postmortem guidance provides a useful basis for recording incidents without blame. Keep unresolved questions visible. Closing the document does not establish that every corrective action has shipped or worked.

    What investigation software can contribute

    A tool can gather and connect evidence that engineers would otherwise retrieve manually. Its usefulness depends on accessible sources, query limits and the quality of its reasoning. It cannot recover telemetry that was never retained or determine an organization's risk tolerance.

    Operate gathers context across connected systems, investigates likely causes and proposes fixes for engineer review. Effective access follows the tools and credentials configured for it. Self-hosting does not by itself enforce read-only permissions, and an external AI provider may receive investigation context.

    To evaluate the workflow, use a resolved incident and compare its output with the evidence your team already has. See production root cause analysis and the deployment boundary. Judge the quality of the explanation, not how confidently it is written.

    About the author

    Operate

    Operate Team

    The team behind Operate

    Operate Team builds self-hosted software for production investigations. Operate gathers evidence from connected systems, investigates likely causes and proposes changes for engineer review. Teams can start with an alert, a recurring check or a question in Slack or Microsoft Teams.

    Share: X LinkedIn
    #Root Cause Analysis
    #Incident Response
    #postmortem
    #SRE
    #Reliability Engineering

    Keep reading