CrashLoopBackOff Triage in the Order That Narrows the Cause

CrashLoopBackOff is the Kubernetes status for a container that has failed repeatedly and is waiting, with a growing delay, before its next restart. We treat it as a description of behaviour rather than a diagnosis, because every cause from a typo to a node problem produces the same word.

Most guides answer it with a long list of possible causes. We find a list less useful than an order, because the right first question removes most of the list in a minute.

In this post we cover what the status means, a four-step triage order, how to get evidence when a container leaves no logs, how to catch a loop that clears itself, and when the order misleads.

CrashLoopBackOff Describes a Restart Loop, Not a Cause

The Kubernetes documentation on the pod lifecycle explains the timing. After containers exit, "the kubelet restarts them with an exponential backoff delay (10s, 20s, 40s, …), that is capped at 300 seconds (5 minutes)." If a container then runs successfully for a while, ten minutes in the documented example, the delay resets.

That timing has two practical effects we keep in mind. A pod looping for twenty minutes has only tried to start a handful of times, and a pod whose cause has already been fixed can still show the status while it waits out its current delay.

The same page lists typical causes, including application errors, configuration errors such as wrong environment variables or missing files, resource constraints, and liveness or startup probes failing. The list is accurate, but it does not tell anyone where to start, which is what the triage order is for.

Triage CrashLoopBackOff in Four Steps

We run these in order during an incident. Each step is quick, and each one rules out a large part of the space before anyone reads a log line.

1. Measure the Blast Radius First

We start with kubectl get pods -A -o wide and look at how many pods are looping, in which namespaces and on which nodes. The answer splits the problem quickly.

• One pod or one deployment, right after a release, points to the new code or its configuration.
• Many workloads across many nodes with no release points to something shared, such as a rotated secret, an expired certificate, DNS or a dependency outage.
• Many pods on one node points to the node itself, such as memory or disk pressure, rather than any application.

We also check the time each loop started. If several workloads began failing within the same minute, we look for the shared event at that minute, such as a certificate expiry, a secret rotation or a node going unhealthy, before opening any single pod.

Getting this wrong is expensive. Teams can spend an hour reading one service's logs when the real cause is a node that every pod on it shares.

2. Partition by Exit Code and Reason

If the problem is local to one workload, kubectl describe pod shows the last termination state, with its exit code and reason. Container exit codes above 128 conventionally mean the process was ended by a signal, with the signal number added to 128.

Exit code	Usual meaning	Where we look next
0	The process finished normally	The restart policy, since a job that completes should not run under Always
1 or other small codes	The application exited with an error	The previous container's logs
127	Command not found, by shell convention	The image entrypoint and command
137	Killed by SIGKILL	The reason field, OOMKilled or not
139	Segmentation fault	Native libraries and the image build
143	Terminated by SIGTERM	Who sent the signal and the shutdown path

Exit code 137 needs the most care. With the reason OOMKilled, the container exceeded its memory limit, which the resource management documentation notes is enforced by the kernel with out of memory kills. Without that reason, something else sent the kill, and we trace that path in our guide to exit code 137.

3. Separate a Crash From a Probe Kill

A container that crashes on its own and a healthy container killed by a failing liveness probe both end in the same loop. The events from kubectl describe pod show probe failures, so we check them before assuming the application is broken.

The pattern is usually visible in timing. A container that dies at the same point in its start-up, just after the probe's initial delay and failure threshold have elapsed, is most likely being killed by the probe rather than crashing by itself. Lengthening the probe without understanding the slow start only hides the problem, so we also ask why start-up became slow.

Slow starts are the classic cause. The documentation on configuring probes recommends a startup probe for these, and explains that "once the startup probe has succeeded once, the liveness probe takes over to provide a fast response to container deadlocks." Our liveness versus readiness probe guide covers probe design in more depth.

4. Check Causes Outside the Container

Sometimes the container is fine and its environment is not. We check four things in turn.

• Secrets and ConfigMaps, where a rotated value or missing key stops the application at start-up.
• Dependencies, because a service designed to fail fast will loop for as long as its database or upstream API is unreachable.
• Init containers, whose failure keeps the main container from starting and whose logs are easy to miss.
• Admission webhooks that mutate the pod, for example by injecting a sidecar or environment that breaks start-up.

These are the cases where reading the application's own code is least useful. The root cause analysis has to follow the dependency or the configuration change outward, and the evidence usually sits in another system's logs or change history.

Getting Evidence When the Container Leaves No Logs

A process that dies before it flushes output leaves kubectl logs empty. The documentation on debugging running pods shows the first fix, reading the previous container's crash log with kubectl logs and the --previous flag.

When that is empty too, the same page describes ephemeral debug containers and debugging a copy of the pod with a changed command, which lets us start a shell in the same environment without the crashing process. We also avoid deleting the deployment to reset it, because that throws away the termination history and exit codes we need.

Seeing how the same status hides different causes in real projects helps here. Browsing root cause analysis run on public GitHub issues shows crash loops traced back to configuration, dependencies and code, often from the same symptom. Image pull failures are a separate status with their own evidence, covered in our guide to fixing ImagePullBackOff.

Catching a Crash Loop That Clears Itself

The hardest loops are the ones that stop before anyone looks. For these we need history rather than current state.

We chart restart counts over time and look for a rhythm. A loop every few hours suggests memory growth, while one at the same time each night suggests a scheduled job or a dependency's maintenance window.

We also record which node each restart happened on and line restarts up against deploys and configuration changes. A loop that only happens on one node pool usually has a cause in that pool, not in the application.

When the Triage Order Misleads

The order assumes the looping workload is the victim of one cause. A few situations break that assumption.

When cluster components such as the network plugin or DNS are themselves crash looping, many unrelated pods fail as a consequence. We fix the platform component first, because application triage will lead nowhere until it is healthy.

A container killed for memory can also be killed again by a probe while it restarts, so the latest termination reason may not be the first cause. We read the earlier terminations in the events before trusting the most recent one.

Finally, an exit code of 0 under a restart policy of Always is not a failure at all. It is a job running in the wrong kind of workload, and no amount of log reading will find a bug.

Start Every CrashLoopBackOff at the Blast Radius

We would make the order a habit on every rota. Count the affected pods and nodes, read the exit code and reason, separate crash from probe kill, check what sits outside the container, and only then dig into logs.

That order turns CrashLoopBackOff from a status into a short list with evidence behind it. Working from evidence is also how Operate investigates, through read-only adapters to logs, databases and repositories, with any code fix proposed as a patch file an engineer reviews and applies.