Kafka Consumer Lag Shapes That Tell You Not to Scale Out

Kafka consumer lag is the gap between the newest offset written to a partition and the last offset a consumer group has committed for it. When it climbs, the reflex is to add consumers, and in several common cases that makes the problem worse.

We see the same scene often. The lag alert fires, the consumers are up, memory is steady, CPU is idling and nothing is logging errors. Low CPU is the clue, because a consumer that is short of compute would be busy.

In this post we cover what lag measures, why to read it per partition, the five shapes and what each one means, a decision table, alerts that match the shapes, and when adding consumers really is the right call.

What Kafka Consumer Lag Measures and What It Misses

Confluent's guide to monitoring consumer lag describes the broker-side metric as "the difference between the last offset stored by the broker and the last committed offset" for each group and partition. Two consequences trip up on-call engineers.

• Committed is not processed. A consumer can finish work on a batch, but until the offset commit succeeds the lag does not move.
• Offsets are not time. Offset lag counts messages, so it stays flat when producers go quiet even if the oldest unprocessed message keeps ageing.
• Messages are not equal. A thousand messages of lag can be seconds of work for a log shipper and hours for a heavy processing job, so one offset threshold rarely fits every group.

That is why we track time lag, the age of the oldest unprocessed message, next to offset lag. Time lag is the number that maps to user impact.

Read Lag per Partition Before Scaling the Group

A single number for the whole group hides the cause. We start every investigation with the per-partition view from the standard tooling.

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group checkout-service

The output lists each partition with its current offset, log-end offset, lag and owning consumer. An illustrative result looks like this.

Partition	Current offset	Log-end offset	Lag	Consumer
0	120500	120500	0	consumer-1
1	98400	105600	7200	consumer-2
2	120500	120500	0	consumer-3

When one partition lags and the rest sit at zero, scaling the group does nothing. That partition already belongs to exactly one consumer, and a new consumer cannot share it.

Five Kafka Consumer Lag Shapes and What Each One Means

We classify lag by how it moves over time across partitions. Each shape has one check that confirms it.

1. All Partitions Rising Together on a Steady Slope

Every partition climbs at the same rate. With high CPU this is a real throughput shortfall. With low CPU the consumer is waiting on something, usually a database, an external API or lock contention, and the confirming check is the latency of that downstream call.

The fix is to speed up or parallelise the downstream work. Adding consumers only helps once the downstream system can take more load, and otherwise it moves the queue into the database, as in the WorkOS pool exhaustion analysis.

2. A Sawtooth That Never Recovers

This is the poll-interval eviction loop. The KafkaConsumer documentation explains that if the consumer does not call poll at least as often as max.poll.interval.ms, "the client will proactively leave the group so that another consumer can take over its partitions."

The arithmetic is simple. With the client defaults of 500 records per poll and a 300,000 ms interval, a consumer that spends 700 ms per record needs 350 seconds per batch. It is evicted before it commits, the batch goes to another consumer, and the cycle repeats. The confirming check is a CommitFailedException in the logs, whose message says the time between polls "was longer than the configured max.poll.interval.ms."

Managed metrics can hide it. Amazon MSK states that consumer lag metrics are emitted only when a group is in a STABLE or EMPTY state, so a group stuck rebalancing can drop off the dashboard entirely. Adding consumers makes each rebalance longer.

3. One Partition Climbing While the Rest Sit at Zero

This is a data problem, not a capacity problem. If the current offset on that partition is not moving at all, a single message is crashing or hanging the consumer, which is a poison message. If it moves but slower than its log-end offset, a hot key is sending that partition far more traffic than the others.

A poison message needs to be moved aside so the partition can progress, which is what a dead letter queue is for. A hot key needs a better partitioning key on the producer side.

4. Flat, Non-Zero Lag While Producers Are Idle

The lag holds at a fixed value even though nothing new is arriving. Either the consumer processed the records but its commits are failing, or the partition belongs to a consumer that has hung without crossing its session timeout. The confirming check is time lag, which keeps rising while offset lag stays flat.

Root cause analysis for these shapes spans the broker, the consumer's logs and the database it writes to. To make that cross-check routine rather than heroic, it helps to connect an investigator read-only to your logs and database before the next incident.

5. Negative Lag or Lag Stuck at One

These are almost always measurement artifacts. On transactional topics, commit markers take up offsets that consumers never receive as records, so a fully caught-up consumer can report a lag of one. Negative values usually come from an exporter reading the committed offset and the log-end offset at slightly different moments.

We confirm by checking if the consumer is actually processing new messages. If it is, we fix the metric rather than the consumer.

A Decision Table for Kafka Consumer Lag

This is the summary we keep in the runbook. The last column matters as much as the fix.

Lag shape	Confirming check	Usual fix	Do not
All partitions, steady climb	Downstream latency and consumer CPU	Speed up or parallelise downstream work	Add consumers before the downstream can cope
Sawtooth that never recovers	Commit failures and rebalance events	Lower records per poll or raise the poll interval	Add consumers, which lengthens rebalances
One partition stuck or slow	Current offset against log-end offset	Move the poison message aside or re-key the producer	Scale the group
Flat lag, idle producers	Time lag rising	Fix failing commits or restart the hung consumer	Trust offset lag alone
Negative or one	Consumer still processing new messages	Fix the metric or exporter	Restart healthy consumers

Alerts That Match the Lag Shapes

A threshold on total group lag catches only the first shape reliably. We add alerts that see the others.

1. Time lag per partition, alerting when the oldest message passes an age the business can tolerate.
2. Rebalance rate, alerting when a group leaves the stable state repeatedly within a short window.
3. Missing lag metrics, alerting when a group's lag series disappears, which on MSK can mean a group stuck rebalancing.
4. Per-partition spread, alerting when one partition's lag diverges sharply from the rest.

When lag drains into a database through an outbox or the reverse, the relay has its own failure modes, which we cover in transactional outbox problems.

When Adding Consumers Really Is the Fix

Scaling out is not always wrong, and we do it without hesitation in one situation. When every partition is climbing, consumer CPU is high, the downstream system has headroom and there are more partitions than consumers, more consumers is the right answer.

Partition count sets the ceiling, though. A group can never usefully run more consumers than the topic has partitions, and adding partitions to an existing topic changes which partition each key maps to. Load testing those limits in production is its own discipline, which Booking.com's approach in capacity tests that expire illustrates.

Newer Kafka versions help with the rebalance cost. The Apache Kafka 4.0 announcement says the project "says goodbye to 'stop-the-world' rebalances" with the general availability of the new consumer group protocol. A consumer that stops polling still gets removed, so the eviction loop does not disappear on its own.

Diagnose Kafka Consumer Lag by Shape Before You Scale

We would read lag per partition, classify its shape, run the one confirming check for that shape, and only then change consumer count or configuration. That order avoids the two cases where scaling makes things worse.

Kafka consumer lag is a symptom whose shape names its cause. When the cause still is not clear, Operate can help find it. It brings the evidence from logs, databases and code together through read-only adapters, and proposes any code fix as a patch file an engineer reviews and applies.