Liveness Probe vs Readiness Probe and What a Restart Deletes

A liveness probe tells the kubelet when to restart a container, and a readiness probe tells Kubernetes when a pod should receive traffic. We treat the liveness probe vs readiness probe choice as a decision about evidence as well as availability, because every restart a liveness probe triggers deletes the state an investigation would need.

Picture a probe failing three times during an incident. The kubelet restarts the container, the alert clears and the pod looks healthy, and nobody can say why it failed because the heap, goroutines and connection pool went with the old process.

In this post we cover what each probe decides, which faults a restart actually clears, a four-question procedure for adding a liveness probe, how to keep evidence when a restart is unavoidable, when the procedure does not apply and a short audit to run this week.

Liveness Probe vs Readiness Probe Decides Restart or Removal

The Kubernetes probes documentation separates three probes by what they trigger. We keep the table below in mind before writing any probe, because the action on failure matters more than the check itself.

Probe	Question it answers	Action on failure	Cost of a false failure
Readiness	Can this pod take traffic now?	Pod removed from Service endpoints	Less capacity until it recovers
Liveness	Is the process stuck for good?	Container restarted	Lost in-memory state and a cold start
Startup	Has the application finished starting?	Other probes held back until it succeeds	Slower detection of a failed start

Readiness failures are cheap and reversible, so we add a readiness probe to almost every service behind a Service or load balancer. Liveness failures are expensive and irreversible for the process, which is why they need a reason.

The same documentation warns that "Incorrect implementation of liveness probes can lead to cascading failures", including restarts under high load and extra work for the remaining pods. A bad readiness probe reduces capacity, while a bad liveness probe can take capacity away exactly when load is highest.

Which Faults a Container Restart Actually Clears

Before configuring a liveness probe, we sort the service's known failure modes into three groups. The group decides whether a restart helps, hides the problem or makes it worse.

1. Cleared for good, such as a deadlock, a wedged event loop or a lock leaked in memory, where the process runs but can never make progress again.
2. Clock reset only, such as a slow memory leak, where a restart removes the symptom and starts the countdown to the next failure.
3. Not cleared at all, such as a slow database, a poison message or bad configuration, where the new container meets the same conditions as the old one.

Only the first group justifies a liveness probe. The second group needs a fix and a trend alert, and a restart can hide that trend from your dashboards for months.

The third group is where liveness probes cause outages. If the probe touches a slow dependency, every replica fails it at once, restarts at once and then fails again, which often ends in a restart loop that our CrashLoopBackOff triage order has to untangle.

Four Questions Before Adding a Liveness Probe

We ask these questions in order and stop at the first "no". Most stateless HTTP services stop at the first question.

1. Can You Name the Stuck State

Name the specific condition the probe is meant to catch, such as a worker pool that deadlocks under a known race. If the process crashes, the container restart policy already restarts it, and if it is slow, the readiness probe already removes it from traffic.

A liveness probe without a named stuck state is a guess. We remove it until someone can describe the failure it guards against.

2. Does a Restart Clear That State

Check the named failure against the three groups above. If it belongs to the second or third group, a restart is the wrong tool, and the probe will add restarts to an already degraded system.

We also ask what the restart costs. A service that takes minutes to warm a cache can lose more availability to one restart than it would to the stuck state.

A useful test is to imagine the probe firing on every replica in the same minute. If the honest answer is that the whole service would restart together and come back to the same conditions, the probe is protecting nothing and adds a second outage on top of the first one.

3. Is Letting It Crash Cheaper

If the application can detect its own fatal state, it can log the reason, write a dump and exit with a non-zero code. The kubelet then restarts it under the restart policy, and the record of why it exited survives in the logs.

We prefer this to an external probe whenever the process can see its own failure. It turns a silent restart into an explained one.

4. Does the Check Stay Inside the Process

A liveness check must never call a database, cache or downstream service. Colin Breck's guide to liveness and readiness probes puts it directly, "Avoid checking dependencies in liveness probes. Liveness probes should be inexpensive and have response times with minimal variance."

Set the timeout above the worst internal latency of the check, including garbage collection pauses. A probe that times out during a long pause will restart a healthy process at the worst moment.

Readiness probes are a different matter. A readiness check may reflect a dependency the pod genuinely cannot serve without, because failing it only removes traffic. We still keep it narrow, checking the one dependency the request path needs rather than every downstream service, so a minor outage elsewhere does not empty the whole Service.

Probe timeouts also show up in application logs in confusing ways. A probe that gives up before the handler answers can appear as a deadline error on the server side, which we explain in context deadline exceeded names the caller.

When a probe does fire, the evidence that explains it is spread across pod events, application logs and the code path the probe exercised. It helps to connect an investigator read-only to your logs and repository so that root cause analysis can start from that evidence before the next restart wipes more of it.

Keep the Evidence When a Restart Is Unavoidable

If a liveness probe is justified, make sure the restart is not the end of the trail. The Kubernetes pod lifecycle documentation notes that "By default, all deletes are graceful within 30 seconds", after which remaining processes receive SIGKILL.

We use that window. A SIGTERM handler can write a thread dump, the in-flight request table and the reason for shutdown to a log that outlives the container, and terminationGracePeriodSeconds can be raised for services that need longer. Work held only in memory dies with the process if the window is too short, as we describe in SIGKILL after thirty seconds.

We also alert on restarts themselves, not only on the probe. Kubernetes events expire quickly, so a restart counter in your metrics is often the only lasting sign a liveness probe fired. When the kill comes from the kubelet after the grace period, the exit code is 137, which we cover in exit code 137 attribution.

When This Liveness Probe Procedure Does Not Apply

Some workloads do have stuck states that only a restart clears, and for them a liveness probe is the right tool. Long-running processes with known deadlock bugs in third-party libraries are the usual example, and so are some language runtimes that can wedge without crashing.

The procedure also matters less where a platform team owns a standard probe library with tested, dependency-free checks. In that case we would still review the timeout and the failure threshold for each service, because a shared default rarely fits every workload.

Startup probes are a separate case. They protect slow-starting containers from liveness kills during start-up, and we add one whenever a liveness probe exists and start-up time varies.

Audit Every Liveness Probe Against a Named Failure

We would run four checks across your deployments this week. For each liveness probe, name the failure it clears, confirm the check touches no dependency, confirm the timeout exceeds the longest pause and confirm the grace period is long enough to write diagnostics.

Any probe that fails the first check is a candidate for removal, leaving readiness to handle traffic. That is the practical answer to liveness probe vs readiness probe for most services. Operate brings evidence from logs, databases and code together through read-only adapters, and every finding cites its evidence.