Why Exit Zero Is Not Evidence a Batch Job Worked

We start with a picture. Imagine a hardening script for a load balancer. It runs, creates an HTTPS listener whose default action returns 403, prints a warning to a log nobody reads and exits with status zero. The automation platform records success and moves on, and the application is unreachable.

The pattern is common in batch work. Background jobs finish without error while repeating the same work, and uptime checks pass while checkout is broken. Every one of those runs passed the only acceptance test it had, which was the exit code.

A Zero Exit Code Describes the Process, Not the Work

In operations we tend to conflate process completion with work validity. An exit status, a 200 response or a completed step in a workflow are statements about the process. They show that the interpreter did not crash and memory was not exhausted.

They do not show that the data was transformed correctly, that a column was not truncated, or that a successful run did not write zero rows because its input was empty. We think that when the work has a known correct answer, the process is not the reliable thing to ask for a status report.

How a Clinical Laboratory Decides a Run Is Acceptable

We find it useful to look at how a clinical laboratory handles a harder version of the same problem. If an analyzer reports a glucose level, the lab cannot trust it just because no error light is on. Labs insert control materials with known values into the same analytical run as patient samples. The controls take the same path, and the run is accepted or rejected on whether the controls land within limits.

The exit status of the instrument is irrelevant if the control comes back wrong. The best-known procedure for judging controls is the set of Westgard rules. The Wikipedia entry on Westgard rules describes them as "a set of statistical patterns, each being unlikely to occur by random variability, thereby raising a suspicion of faulty accuracy or precision."

The False-Alarm Mathematics Engineers Need

The usual objection to adding checks is noise. Everyone has met the threshold alert that fires every Sunday because traffic is low, and then gets muted. The laboratory field quantified this trade-off decades ago, and we think software teams can borrow the numbers.

The Westgard multirule page reports that a simple 1-2s rule, which rejects a run when one control is more than two standard deviations from the mean, has a false rejection rate that grows with the number of controls.

Controls per run (N)	False rejection of good runs by the 1-2s rule
1	About 5%
2	About 9%
3	About 14%
4	About 18%

The laboratory answer is not a better single threshold. It is a multirule procedure, in which the 1-2s rule serves only as a warning. The Westgard site puts it this way, "In manual applications, a 1-2s rule should be used as a warning to trigger application of the other rules."

• 1-3s. One control exceeds three standard deviations.
• 2-2s. Two consecutive controls exceed the same two standard deviation limit.
• R-4s. One control exceeds plus two standard deviations while another exceeds minus two.
• 4-1s. Four consecutive controls exceed the same one standard deviation limit.
• 10x. Ten consecutive controls fall on the same side of the mean.

Key Takeaway. Set an explicit target for how many bad runs you must catch and how many good runs you are willing to reject, and write that down before you write the monitoring.

What Transfers and What Does Not

We should state the analogy narrowly. Most software batch outputs are not stable, continuous quantities like a chemical analyte.

What transfers is the control record, which is a known-answer record pushed through the real path of the job on every run. Multirule logic also transfers, meaning more than one rule, including rules that look back across previous runs to find drift. So does an explicit false alarm budget.

What does not transfer is the sigma constants. The two and three standard deviation limits assume a roughly normal distribution that many database writes will not follow, so we would use fixed invariants or percentiles. Stop-the-line cost differs as well, since software can retry or divert work to a dead-letter queue.

Designing a Control Record for a Real Job

A control record is a synthetic input that travels the same code path as real work without polluting production metrics. The table below pairs common failure classes with a check on the control and a rule shape.

Failure class	What to check on the control	Rule shape
Partial write	Exact value of the control record	Single run, strict
Duplicate write	Unique constraint or count on the control ID	Single run, strict
Silent truncation	Length or checksum of a control field	Single run, strict
Empty input	Presence of the control in the output	Single run, strict
Stale input or drift	Latency timestamp on the control record	Four runs in a row

For a billing job, the control might be a test account for which the job must compute exactly ten dollars. If the output shows zero or a rounding error, we reject the run even though the commit succeeded. Our guide to pipelines that stay green and deploy nothing lists related failure shapes that a zero exit hides.

Where the Control Sample Approach Fails

Some environments prohibit synthetic data, such as core ledgers under audit rules. There the honest answer is post-hoc reconciliation or read-path checks, not in-process control records.

The approach also fails quietly when controls are not updated with schema changes, because the quality control itself then becomes a source of false negatives. We would put the control record under the same review as the job it guards, and give it an owner.

A Failed Control Needs an Investigation

A control that fails tells you a run is bad and rarely tells you why. The cause might be malformed input, a hidden schema change, a changed query plan or unstable infrastructure. Google's SRE book offers a relevant rule in its chapter on monitoring distributed systems, which says every page should be actionable. A rejected run that arrives with no evidence is not.

The gap between a rejected run and the evidence for a fix is where we would use dedicated AI agents that investigate with cited evidence, while a person decides. Our notes on detecting silent failures from outside the system cover the wider method.

The Two Numbers to Put on the Wall

A team starting out needs two numbers. The first is the share of bad runs the controls would have rejected. The second is the share of good runs they actually rejected.

Everything else, including the multirules, the synthetic records and the drift checks, is a way to make those two numbers acceptable. For a related method, see our piece on silent failures in agent pipelines that return 200 OK.

Stop Trusting Exit Zero and Start Trusting Controls

Exit zero is a statement about a process. A control record is evidence about the work, and a rule set tuned for false alarms keeps the evidence usable.

We would add one control record to your most important batch job this week and track the two numbers. When you want help investigating failed controls, Operate runs dedicated AI agents that investigate production problems with cited evidence and propose fixes for your team to review.