SIGKILL After Thirty Seconds Erases Scheduled Work in Memory

SIGKILL is the signal that ends a process immediately, with no chance to clean up or save state. In Kubernetes it arrives after every deploy, scale-down and eviction for any process that is still running when the grace period ends.

We think the most expensive thing it destroys is not in-flight requests but scheduled work. A reminder set to fire in four hours lives only in the memory of the process that set it, and when that process is replaced at lunchtime the reminder never happens.

In this post we cover why this failure leaves no evidence, where in-memory scheduling hides, what happens between SIGTERM and SIGKILL, why the autoscaler cannot help, why replicas create duplicates, the limits of CronJobs and Celery, and what durable scheduling requires.

The Failure That Leaves No Evidence Behind

Most production failures deposit evidence. An unhandled exception leaves a stack trace, a memory leak ends in an OOM kill, and a failing dependency produces 5xx responses or dead-letter messages.

This one deposits an absence. Nothing throws, because the scheduled work was simply never run, and the first signal is usually a customer asking why an email or a renewal never arrived.

That makes it a silent failure in the strict sense. It cannot be found by watching error rates, only by checking that work which should have happened did happen, the method we describe in detecting silent failures from outside.

Where In-Memory Scheduling Hides in Ordinary Code

In-memory scheduling is everywhere because it is one line of code. We look for it in these places first.

• JavaScript setTimeout and setInterval, and libraries such as node-cron.
• Spring's @Scheduled annotation and other in-process schedulers on the JVM.
• Python's APScheduler with an in-memory job store, and Celery tasks sent with eta or countdown.
• Go's time.AfterFunc and goroutines that sleep before doing work.
• Any retry loop that sleeps for minutes inside a request handler or worker.

What they share is that the schedule exists only in one process's memory. If the process dies, the schedule dies with it.

What Happens Between SIGTERM and SIGKILL When a Pod Stops

The pod lifecycle docs describe a precise sequence. It runs on every rolling deploy, every scale-down and every voluntary eviction.

1. The Container Is Asked to Stop

The kubelet asks the runtime to stop each container, and the runtime sends SIGTERM unless the image defines a different STOPSIGNAL. If a preStop hook is configured, it runs first, inside the same grace period rather than in addition to it.

2. The Grace Period Runs, Thirty Seconds by Default

The docs state that "the default terminationGracePeriodSeconds setting is 30 seconds." A preStop hook still running when it expires gets a one-off extension of 2 seconds, and nothing more. Thirty seconds is sized for draining in-flight requests, not for finishing work scheduled hours ahead.

3. SIGKILL Ends Everything Still Running

"Once the grace period has expired, the KILL signal is sent to any remaining processes." Because SIGKILL cannot be caught or handled, no shutdown hook runs, and every timer, queue and schedule held in that process's memory disappears.

A clean SIGTERM handler does not save scheduled work either. It can stop accepting new requests and finish current ones, but it cannot run a job due in four hours. In Node.js, installing a SIGTERM listener does not even keep the process alive if the event loop has nothing left to do.

When a lost job is finally noticed, finding out which deploy or scale-down killed its process means correlating pod events, application logs and the job's own records. It helps to connect an investigator read-only to your logs and database before that question comes up, because Kubernetes events expire quickly.

The same signal produces exit code 137, and working out which component sent it is a separate investigation, which we cover in exit code 137 attribution.

The Autoscaler Does Not Know What the Pod Was Holding

We often hear the assumption that the autoscaler avoids removing busy pods. It does not. The Horizontal Pod Autoscaler changes a replica count, and the ReplicaSet controller chooses which pods to delete.

The ReplicaSet docs list the order. Pending pods go first, then pods with a lower pod-deletion-cost, then pods on nodes with more replicas, then the newer pods, and "if all of the above match, then selection is random."

The deletion-cost annotation is the only lever, and the docs warn it "is honored on a best-effort basis, so it does not offer any guarantees on pod deletion order." A pod cannot make itself safe by holding work in memory. The only fix is to stop holding it there.

More Replicas Turn Lost Jobs Into Duplicate Jobs

When jobs go missing, the first instinct is often to add replicas for safety. With an in-process scheduler, three replicas do not make the job highly available. They make it run three times, once in each replica.

Every replica runs its own copy of the schedule unless something coordinates them. Libraries such as ShedLock and Quartz exist largely to add that coordination, using a shared lock or job store so that only one instance runs each execution.

So adding replicas, as we put it to teams, trades one failure for another. Lost work becomes duplicated work, which is only harmless if every job is idempotent.

Moving Work Out of the Process Is Necessary but Not Sufficient

External schedulers help, but each has its own rules. The CronJob docs say scheduling is approximate, because "there are certain circumstances where two Jobs might be created, or no Job might be created," and conclude that jobs should be idempotent. The default concurrencyPolicy is Allow, and a CronJob that misses more than 100 schedules stops starting the job and logs "too many missed start times."

Celery's eta and countdown carry their own warning in the Celery calling guide. Such tasks "are immediately fetched by the worker and until the scheduled time passes, they reside in the worker's memory." The guide recommends against them for scheduling far in the future and suggests database-backed periodic tasks instead.

Instawork's engineering team reached a similar conclusion after studying Celery's internals, writing that countdown and ETA should be used rarely. The pattern we see is consistent. Anything that parks future work inside a running process inherits that process's lifetime.

What Durable Scheduled Work Requires

Work survives a deploy only when its existence is recorded outside the process that will do it. We look for four properties.

1. The intent is written durably, in the same transaction as the state change that created it, which is the job of a transactional outbox.
2. Execution is leased, so a worker that dies mid-job releases it for another worker to claim, for example with SELECT ... FOR UPDATE SKIP LOCKED or a queue visibility timeout.
3. Jobs are idempotent, because leases and retries mean some work will run twice, as we argue in exactly-once is a property of the write.
4. Missing work raises an alert, because this failure produces no errors to alert on.

GOV.UK's email platform shows the recovery side of this. Its architecture decision record on lost Sidekiq jobs notes that jobs were lost "frequently due to workers being forcibly restarted when they consume too much memory," including a Travel Advice change that led to an incident. The team added a worker that periodically requeues work not processed in time.

When In-Memory Scheduling Is Acceptable

We do not ban every timer. Short timers that fire well inside the grace period, such as debouncing a burst of events or retrying a call after a second, rarely lose anything that matters.

Work that is re-derived on startup is also safe. A cache refresh or a metrics flush that the next process will simply redo does not need durable scheduling, because losing one run costs nothing.

The test is the question an auditor would ask. If this process disappeared right now, would anything else know this work was supposed to happen? If the answer is yes, or if the work does not matter, a timer is fine.

Audit Every Timer Before the Next SIGKILL Arrives

We would search the codebase for each language's scheduling calls, ask that question of every hit, move the ones that matter to durable storage with leases, and add an alert on work that stops completing. Root cause analysis of a lost job is far easier once the intent was recorded somewhere.

SIGKILL will keep arriving on every deploy, so scheduled work has to outlive it. When a lost job still needs explaining, Operate can help trace it. It brings the evidence from logs, databases and code together through read-only adapters, and proposes any code fix as a patch file an engineer reviews and applies.