Exit Code 137 Does Not Mean Out of Memory: Proving Which Killer Sent SIGKILL

TL;DR: Exit code 137 means a process was terminated by a SIGKILL (signal 9), calculated as 128 plus the signal number. While often associated with OOMKilled events, it is also triggered by kubelet evictions, liveness probe failures, graceful shutdown timeouts, and external CI runner cancels.

The number only tells you a signal arrived

When a container or process dies and reports exit code 137, you are looking at simple addition: 128 + 9 = 137. In Linux systems, exit codes above 128 indicate that the process was terminated by a signal. Signal 9 is SIGKILL, the "hard kill" that cannot be caught, blocked, or ignored by the application.

What exit code 137 tells you is that something sent SIGKILL and the process had no choice but to stop immediately. What it does not tell you is which component of the system sent that signal, or why.

You will often see sibling codes in your logs:

• Exit code 143 (128 + 15) represents SIGTERM, a graceful request to stop.
• Exit code 139 (128 + 11) indicates a segmentation fault (SIGSEGV).

If you see 143, your process was asked to leave; if you see 137, your process was thrown out. Note that in rare cases, a process can explicitly call exit(137) itself, meaning the number alone is not absolute proof of an external signal, though it is the primary indicator in containerized environments.

Key Takeaway: Exit code 137 is a symptom of a SIGKILL termination, not a diagnosis of memory exhaustion, and requires cross-referencing system logs to identify the true killer.

Six things produce exit code 137

The reason exit code 137 kubernetes and exit code 137 docker searches are so common is that multiple system components use SIGKILL as their final lever.

1. cgroup OOM kill: The container exceeded the specific memory limit defined in its manifest. The kernel kills a process within that specific control group (cgroup).
2. Node-level kernel OOM kill: The entire host ran out of memory. The Linux kernel's OOM killer picks a victim based on oom_score. This might be your container even if your container is under its own limit.
3. Kubelet eviction: Under disk or memory pressure, Kubernetes makes a scheduling decision to reclaim the node. It terminates the pod to move it elsewhere.
4. Graceful shutdown timeout: The system sent a SIGTERM (code 143), but the application hung or took too long to clean up. After the terminationGracePeriodSeconds expires, the runtime escalates to SIGKILL.
5. Liveness or startup probe failure: The orchestrator determines the container is unhealthy. It initiates a restart, which follows the same escalation path from SIGTERM to SIGKILL if the process doesn't exit promptly.
6. External kill: A CI/CD harness (like GitHub Actions) times out a job, a script runs docker kill, or an operator manually executes kill -9.

Only the first two are strictly "out of memory" events, and only the first one reliably sets the OOMKilled flag to true.

The attribution table: Finding the evidence

Because exit code 137 is ambiguous, you must look for the specific artifacts left by each killer.

The procedure, ordered by what it rules out fastest

To diagnose a command failed with exit code 137 error, follow this sequence:

1. Anchor on the timestamp: Find the exact second the container died and its unique Container ID. Evidence in system logs is useless without a precise time-window.
2. Check Container Status: Run kubectl describe pod or docker inspect. If OOMKilled is true, your container hit its limit. If it is false, you have proved nothing; move to step 3.
3. Rule out Orchestrator Kills: Look for Unhealthy (probes) or Evicted events. These explain why the system chose to kill the container.
4. Check the Node Logs: If the orchestrator is silent, check the host kernel logs (dmesg -T | grep -i oom). If you see "Out of memory: Kill process," the host was under pressure.
5. Audit CI/Harness Logs: If this happened during a build (e.g., exit code 137 docker build), check if the CI runner itself was terminated by the platform for exceeding job duration limits.

Why OOMKilled is false so often

The oomkilled meaning in container runtimes is specific: it means the cgroup manager successfully attributed the kill to a limit breach.

However, oomkilled is often false even when memory is the culprit. This happens if the host kernel kills the process before the cgroup limit is hit, or if a child process inside the container is killed while the main PID 1 remains alive. If you see linux exit code 137 without the OOM flag, it usually means the killer came from outside the container's own resource constraints.

The JVM and other runtimes that hide their own limits

For python exit code 137 or java exit code 137 cases, there is a nuance in how runtimes handle memory. According to [specificlanguages.com], if the JVM hits its internal -Xmx limit, it will usually throw an OutOfMemoryError and exit with a code like 1. If it exits with 137, the OS or container runtime killed it before the JVM even realized it was in trouble. This happens when your container memory limit is set too close to your JVM heap size, leaving no room for metaspace, stacks, and native overhead.

The evidence expires, usually before anyone looks

The greatest challenge with kubernetes exit code 137 is that the evidence is ephemeral. Pod objects are replaced on restart, clearing the "Last State." Kernel ring buffers rotate under high log volume. Ephemeral CI runners vanish the moment the job fails.

This is why this matters: For intermittent 137 errors, the first occurrence is often the only one with enough context to diagnose. By the time an SRE investigates the third occurrence, the logs that proved a node-level OOM or a probe timeout are often gone.

Fixes, matched to cause rather than to the number

• For cgroup OOM: Increase the memory limits in your manifest.
• For Node OOM: Spread your workload using pod anti-affinity or add more nodes to the cluster.
• For Graceful Timeout: Increase the terminationGracePeriodSeconds or optimize your app's SIGTERM handler.
• For Probe Failures: Check application health or increase the failureThreshold and initialDelaySeconds.

Sources & further reading

• According to [specificlanguages.com], exit 137 in CI often points to a mismatch between runner memory and build tool requirements.
• According to [oneuptime.com], the 10-second default graceful shutdown period in Docker is a frequent source of "false" 137 errors.
• According to [groundcover.com], node-level memory pressure can trigger 137 even if a pod's individual usage is low.

Automated Attribution with Operate

The recurring exit code 137 is a classic example of "lost context" in production. Because the deciding evidence—cgroup events, kernel logs, and orchestrator signals—vanishes so quickly, engineering teams often resort to guessing and raising memory limits as a reflex.

Operate solves this by capturing the state of the node, orchestrator, and runtime at the exact moment of termination. When a 137 occurs, Operate investigates the incident, finds the root cause with evidence (like correlating a probe failure to the SIGKILL), and drafts a fix. This ensures that when an engineer looks at a failure, the evidence hasn't already expired.

FAQ

What does exit code 137 mean? It means the process received a SIGKILL (signal 9) and was forced to terminate immediately.

Does exit code 137 always mean out of memory? No. It can also be caused by probe failures, graceful shutdown timeouts, or manual process termination.

What is the difference between exit code 137 and 143? 143 is a graceful termination (SIGTERM), while 137 is a forced termination (SIGKILL).

Why is OOMKilled false when the exit code is 137? This happens if the process was killed by something other than the container's memory limit, such as a node-level OOM killer or a timeout.

How do I prove which process sent SIGKILL? You must check the Kubernetes events for Killing messages, the kernel logs for oom-kill entries, and the container runtime logs for timeout expirations.