Interviewing for Diagnosis Versus Construction in SRE Hiring

Interviewing for diagnosis is rare, and we think the gap is worth looking at. Hiring loops spend hundreds of hours testing the ability to build, while teams pay a daily cost for the ability to work out why something broke.

In this essay we compare what Google's SRE book says about training and selection, what the selection research supports, how to design a diagnosis round and where its limits are. We state plainly where the evidence stops.

The Round That Is Not in Your Loop

Imagine a production incident that defies the runbook. The person who finds the answer rarely does it by writing an elegant algorithm. They rule things out, isolate variables, question their own assumptions and build a mental model of a system they did not write.

Now look at your hiring loop and ask which stage would have detected that ability. For most organisations the honest answer is none of them. We would rather name that gap than assume coding skill covers it.

Google Says Both Things in the Same Book

We find the SRE book holds an internal tension. Its chapter on effective troubleshooting says that troubleshooting is "both learnable and teachable", and describes the method as iteratively hypothesising causes and testing them.

Its chapter on accelerating SREs to on-call sets out the training. It lists Wheel of Misfortune role-play, a class on reverse engineering a production service, and an exercise in which a group cripples a search cluster stack at its bottlenecks. The attributes it aims to build are reverse engineering skills, statistical thinking and improvisation when procedures fail.

The introduction describes the hiring side differently. "50–60% are Google Software Engineers, or more precisely, people who have been hired via the standard procedure for Google Software Engineers." The other 40 to 50 percent are close to those qualifications and bring skills such as UNIX system internals and networking.

That is a coherent position. If a skill is teachable, you train for it instead of narrowing the hiring pool to find it. The question for other teams is if they run the training half, or only skip the selection half.

What Diagnosis Costs You Every Working Day

The cost of the gap is hard to measure precisely, and the survey data we have is self-reported. The 2024 Stack Overflow survey reports that technical debt is the top frustration at work for professional developers, at 63% of respondents. It also reports that 61% of respondents spend more than 30 minutes a day searching for answers or solutions to problems.

We do not read that as a measurement of diagnosis time. It does suggest that searching for the cause of problems takes up a noticeable share of every working day, which is the work a coding round does not sample.

The Evidence That Does Not Exist

A common defence of coding-only loops is that coding ability and debugging ability go together. We looked for a study showing it and did not find one. We are not claiming that none exists, only that we could not find a study that links coding interview performance to the ability to diagnose production failures.

Key Takeaway. We found no evidence that high scores in algorithmic coding interviews predict the ability to diagnose complex production failures, and no evidence against it either.

That leaves a judgement call, and it should be made openly. A diagnosis round is a bet that sampling the real work beats assuming that coding skill transfers.

What the Selection Research Actually Supports

For decades the textbook position gave general cognitive ability the leading role. A 2022 paper in the Journal of Applied Psychology, Revisiting meta-analytic estimates of validity in personnel selection, argues that earlier range-restriction corrections caused substantial overcorrection. Its abstract reports revised validity estimates reduced by .10 to .20 points and says "Structured interviews emerged as the top-ranked selection procedure."

We take two things from this. Selection procedures remain useful but less predictive than once thought, and a structured interview with defined scoring beats an unstructured conversation. A debugging interview with a clear rubric can be run as a structured interview. We do not claim the paper tests debugging rounds, and the abstract does not.

What Interviewing for Diagnosis Looks Like in Practice

A diagnosis round puts a candidate next to an interviewer with a real historical bug, ideally from an open-source project, using their own tools. Some large companies are reported to run a round of this kind, and we could not verify details from a primary source, so we describe the format generically.

What it can score is the ability to form hypotheses, to build a mental model of unfamiliar code and to explain the path to an answer. We expect that last part to matter more as candidates use AI tools, because the explanation of why is harder to borrow than the fix.

The costs are real. Authors have to write and maintain realistic bugs, calibrate their difficulty, prepare several to limit leaks, and keep the snapshot building as dependencies age. We would not add the round unless we could staff that upkeep.

Designing the Round With an SRE Rubric

If you add a diagnosis round, keep it from becoming trivia about your own codebase. Use a real historical bug and score the search, not the answer. The SRE book's troubleshooting chapter lists common pitfalls that work as a rubric.

• Irrelevant symptoms. Does the candidate get pulled toward noise in the logs or metrics?
• Misunderstanding how to test. Do they change the system safely when checking a hypothesis?
• Improbable theories. Do they reach for exotic explanations before checking the configuration?
• Latching onto past causes. Are they solving the last incident instead of this one?

The same standard applies to tools. When a platform claims to have found a root cause, we want it to show its evidence in the way we ask of a candidate. That is the discipline behind dedicated AI agents that cite their evidence, with a person reviewing the conclusion. Our framework on when to hire an SRE covers what should exist before the hire.

What a Diagnosis Round Cannot Prove

We should be honest about the limits. One interview samples one bug on one day, and a candidate who is brilliant with logs may be weak at diagnosing a database. A strong score is evidence of a skill and not a guarantee of performance on call.

It also cannot replace the training half. Google's approach assumes that you teach troubleshooting after you hire, and a diagnosis round only changes who arrives with a head start. Our notes on on-call onboarding and time to first page describe the training side, and our piece on runbook automation and diagnosis explains why diagnosis stays a human skill longer than execution.

Hold Hiring and Tools to the Same Diagnostic Standard

Interviewing for diagnosis tests if an engineer can separate an observation from a conclusion under pressure. That is the core discipline of the job, and we would sample it instead of assuming it.

SRE hiring is a place where the standard can start small. We would start by adding one structured question about a past incident to an existing round, and score the reasoning. If you want help gathering evidence while your engineers keep the verdict, Operate runs dedicated AI agents that investigate production problems with cited evidence and propose fixes for your team to review.