Silent Failure Detection Needs an Expectation Held Outside

A silent failure is the absence of an expected effect with no error to show for it. A webhook processor stops, a job never runs or a queue drops items, and every dashboard stays green because nothing reported a problem.

We think the usual advice, which is to add more logging, fails for a structural reason and not for lack of effort. That reason decides what kind of detector can work, and it is where silent failure detection should start.

In this post we cover why logging cannot catch an absence, the one principle that can, the four shapes an external expectation takes, who watches the watcher, why these checks rarely get built, a checklist to start this week and when the effort is not worth it.

Why Adding More Logging Cannot Catch a Silent Failure

Logging is an act of emission. It needs the component to be alive, aware of its own state and able to write something down. A silent failure is defined by the opposite condition, and absence is not an event that anything emits.

A worker that has crashed does not log that it stopped, because the process is gone. A swallowed exception does not raise, since the code treated it as a successful no-op. A scheduled job that never gets scheduled produces no line, because the mechanism that would have written it never ran.

If we try to detect the failure of a component from inside that component, we are asking the failing part to report on itself. That is a category error rather than a maturity gap, and no amount of extra log statements closes it.

Google's SRE book makes a related point in its chapter on monitoring. It defines black-box monitoring as "Testing externally visible behavior as a user would see it" and says it is "symptom-oriented", covering problems that are active rather than predicted. The observer sits outside, and that position is what lets it notice when something stops.

An Expectation With a Deadline Is the Only Detector of Absence

To catch an absence we need two things together. The expectation must live outside the component, and it must be attached to a clock, because without a deadline an expectation is only a hope.

The external position matters for a second reason. A system that certifies its own success has a conflict of interest, and a green status from the thing being measured is weak evidence.

Tools built around this idea are simple. Healthchecks.io, for example, "listens for HTTP requests ("pings") from your cron jobs and scheduled tasks", "keeps silent as long as pings arrive on time" and "raises an alert as soon as a ping does not arrive on time." The alert comes from the watcher noticing a missing event, not from the job reporting anything.

Prometheus offers the same logic for metrics. Its query functions documentation says absent() returns "a 1-element vector with the value 1 if the vector passed to it has no elements", which it describes as "useful for alerting on when no time series exist for a given metric name and label combination."

Four Shapes an External Expectation Can Take and What Each Catches

Most missing-event incidents can be caught by one of four structures. They differ in cost and in what they cannot see, which is why we pick by failure mode instead of defaulting to the cheapest.

Shape	Expectation	Catches	Blind spot
Deadline on an expected event	A heartbeat or freshness check must arrive within a set time	A stopped processor or a job that never ran	A process that runs on time and does nothing
Reconciliation	Declared state must match observed state	A half-configured system behind a green status	Drift in what was declared
Conservation invariant	Work accepted equals work done plus failed plus in flight	A queue that drops items without a trace	Work that is counted twice or miscounted
Second independent signal	Two instruments agree about the same event	A success status with the wrong content	Both instruments sharing a cause

1. The Deadline on an Expected Event

This is the heartbeat, or dead man's switch. A table of last successful runs, a freshness check on a timestamp, a ping per run or an alert on missing metrics all use it. It is cheap, and it catches a very common case, which is that something stopped.

Its blind spot is the zombie. A process that runs on schedule but does no real work satisfies a deadline check perfectly, so for important jobs we pair the heartbeat with a check on what the job produced.

2. The Reconciliation

Kubernetes uses this idea throughout. Its controller documentation says "controllers are control loops that watch the state of your cluster, then make or request changes where needed", each trying "to move the current cluster state closer to the desired state."

The same structure works for detection. Declare what should be true of a server, a deployment or a pipeline, and let a separate process compare it with what is. It costs the most to build and gives the strongest protection against a half-finished change reported as success.

3. The Conservation Invariant

For anything that moves work, the arithmetic is simple. What was accepted must equal what completed, plus what failed, plus what is still in flight. A gap means items vanished.

This catches the queue that quietly loses messages without a dead letter record. We discuss the handling side in our dead letter queue operating manual, and the write-side guarantee in the transactional outbox guide.

4. The Second Independent Signal

Two instruments that observe the same event from different angles disagree when one is wrong, and the disagreement is the alert. A request can return success while the content is wrong, which a status code alone cannot show.

It is increasingly relevant for AI-driven pipelines, where a healthy response says little about whether the answer was right, as we describe in silent failures in AI agent pipelines. The blind spot is shared cause, so the signals have to be genuinely independent.

Who Watches the Watcher Without an Infinite Chain

The obvious objection is regress. If a heartbeat monitor can fail, something has to watch the monitor, and that something needs watching too.

We do not think the answer is an endless chain. It is a short chain in which each link fails differently, for example a heartbeat service outside your own infrastructure that expects a ping from your alerting pipeline, so that a dead alerting path becomes a missed deadline somewhere else.

We then test the whole arrangement by breaking things on purpose. Stop the job, drop the message, kill the alerting path, and confirm that someone is told. If we have never tested the absence, we have not solved the silent failure, only moved it.

Why External Expectations Rarely Get Built

The technical path is clear, so the gap is usually organisational. An expectation held outside a component often falls outside the owning team's boundary, which can make it nobody's ticket.

The team that owns the webhook processor may have no roadmap room for a cross-service heartbeat, and the platform team does not feel the pain of the failure. In our reading, that is why people reach for logging, since it is the one tool they already own, and it is also the one tool that cannot do this job.

A Starting Checklist for This Week

We would pick the single most important scheduled path you have and answer these questions about it, in writing.

• What is the longest interval a scheduled job could be dead before a human would notice?
• Which queues can lose an item without producing a dead letter record?
• Which deployments report success based only on the exit code of the script that ran them?
• Is there a work-in against work-out view for the most critical pipeline?
• Which of your alerts would stay silent if the alerting pipeline itself failed?

When one of these expectations fires, the work shifts to finding the cause, and root cause analysis starts from the missed deadline. That is where an AI agent that investigates with evidence can help, by gathering the logs, database state and recent changes around the missed deadline for an engineer to review.

Related reading on the failure itself includes scheduled work that is killed after thirty seconds, a common reason for jobs that vanish.

When an External Expectation Is Not Worth Building

Not every job deserves a heartbeat. A task that runs every few seconds and is harmless to skip does not justify a separate watcher, and a failure that is loud already has a detector in the error path.

Cost matters too. Reconciliation and conservation checks need design effort, and we would not build them for a low-value internal job. The test we apply is what the first missed run would cost and how long it would stay undetected.

Also, an external expectation is only as good as its deadline. A deadline set so loosely that it fires after the damage is done gives false comfort, so we set it from the longest acceptable gap, not from the usual run time.

Start Silent Failure Detection With the Job Nobody Watches

We would choose the path whose failure would go unnoticed longest, state what should have happened and by when, and put that expectation outside the thing it checks. Then test it by causing the failure on purpose.

Silent failure detection is mostly a decision about who holds the clock. Operate reads logs, databases and repositories through read-only adapters, and proposes any code fix as a patch file an engineer reviews and applies.