Booking.com's Kafka capacity tests expire on their own rather than being cleaned up afterwards. That design rule applies to every automated change you make to production.

Booking.com's Capacity Test Cannot Be Left Switched On: Every Command It Sends Expires By Itself

TL;DR: Booking.com’s Kafka capacity testing framework uses time-to-live (TTL) leases for every production change, ensuring that automated experiments revert to safety if the controller fails. This "self-expiring" design prevents stuck production states and proves N-1 resilience without requiring manual, destructive failure drills.

To maintain high availability, engineering teams must answer a difficult question: If one of N instances fails, can the remaining N-1 carry the load without breaching health limits? Traditionally, teams answer this by performing a manual drill: they kill an instance in production and watch what happens.

This approach is fundamentally flawed. It reproduces the right failure mode by causing a real one. It offers no controlled ramp-up, lacks a consistent stopping rule, and provides no automatic recovery if the engineer running the test is pulled away or loses their connection.

Kafka has no traffic knob

For a standard HTTP service, load testing in production is conceptually simple: you adjust a load balancer to send a higher ratio of traffic to a specific target. Kafka consumers do not work this way. In an event-driven system, load is determined by partition assignment.

According to Kaan Karakaya at Booking.com Engineering, the team solved this by building a custom partition assignor. Instead of a standard round-robin or range assignment, their system allows a scheduler to give a target instance "priority." This forces the instance to receive more partitions than its peers, concentrating the load exactly as a failure would, but without terminating a healthy process. This allows for precise capacity testing by shifting work rather than destroying capacity.

The part worth copying: the lease

The most critical design choice in Booking.com’s system isn't how it moves the partitions, but how it stops moving them. Every scale command issued by their capacity API carries a time-to-live (TTL). The client library, residing in the consumer itself, only honors the temporary "high-load" assignment while that lease is valid.

Key Takeaway: An automated change to production must expire on its own, because an undo that depends on the thing that made the change is not a reliable undo.

A successful test ends with an explicit reset command. However, if the test scheduler crashes or the network partitions mid-test, the lease simply expires. The consumer then reverts to its normal assignment logic automatically. This inverts the typical failure mode of automation, where a controller that dies halfway leaves production in a "zombie" state that nobody chose and nobody is watching.

This principle of bounded state is why Operate takes a similar posture toward production changes. By proposing changes as PRs for human review rather than holding direct write access, the system ensures that every change is bounded by a deliberate human action rather than an open-ended automated permission.

Everything in your stack that has no lease

Most production environments are littered with "unleased" state. When we change production, we usually plan a cleanup step that never happens. Consider these common scenarios:

Feature Flags: An experiment concludes, but the flag remains in the database for months because the "cleanup" ticket was deprioritized.
Canary Weights: A deployment pauses at 30%, the CI runner times out, and the traffic split stays lopsided indefinitely.
Debug Logs: An incident requires DEBUG level logging. The incident ends, but the increased disk I/O and cost persist because the engineer forgot to toggle it back.
Rate-Limit Overrides: A VIP customer needs a temporary quota increase that becomes a permanent vulnerability.

Implementing a lease for these is not always easy. It requires the target system (the flag evaluator, the logger, the IAM provider) to understand time. But without it, you are always one crashed script away from a permanent production drift.

Health is not a 200 OK

Booking.com’s framework rejects the idea of a single "up/down" success signal. Instead, they use a composite gate to determine if a capacity step passed. To avoid kafka consumer lag, they don't just check if the service is alive; they monitor:

CPU and Memory: Measured against safety limits that include deliberate headroom (e.g., 80% is a fail, even if the process is running).
Record Age: A configured percentile of how long a message has existed before being processed.
Error Rates: Throughput is meaningless if the service is just NACKing every message.

The generalization here is vital: capacity is whichever constraint binds first. If you only instrument CPU, you will eventually be blinded by a memory leak or a disk I/O bottleneck that hits its limit while CPU sits at 20%.

Report the effective number, not the requested one

In Kafka, partition counts are integers. If you have five partitions, the smallest load increment you can apply to a single instance is 20% of the total topic traffic. You cannot ask for a 5% increase.

Booking.com’s system is honest about this physical constraint. It reports the effective ratio—what actually happened—rather than the requested ratio. Most internal tooling does the opposite, showing a smooth requested ramp on a graph that masks the reality of a discrete, "chunky" load distribution. When debugging consumer lag later, that discrepancy between the graph and the reality can lead to hours of wasted investigation.

What this buys an engineering leader

When capacity testing is automated and self-healing, N-1 resilience stops being an aspiration in a design doc and becomes a verifiable metric. It allows for proactive kafka consumer lag monitoring that identifies bottlenecks before a real outage occurs.

Right-sizing infrastructure stops being a finger-in-the-wind exercise based on peak-load guesses. Instead, you get evidence that is comparable over time. You can see exactly how a new code release changed the "breaking point" of your consumers, providing the auditability and control necessary for high-stakes production environments.

What it does not buy

Automation changes the quality of the evidence, but it does not change the constraints of the system. As Booking.com notes, partition counts still dictate your testing resolution. Rebalances are still group-wide events, meaning you cannot run these tests on multiple consumers in the same group simultaneously. Furthermore, health signals still have inherent observation delays. You aren't avoiding the laws of physics; you're just making sure you don't break production while trying to understand them.

Sources & further reading

According to Kaan Karakaya (Booking.com Engineering), automated capacity testing for Kafka requires a custom partition assignor to concentrate load without service interruption.
Data from Ubersuggest indicates that while "capacity testing" is a high-volume search term, it is frequently conflated with medical and industrial testing, making precise engineering case studies like Booking.com's essential for practitioner clarity.
The Booking.com Development blog highlights that composite health checks—combining resource headroom with lag percentiles—are the only way to accurately define service capacity.