Low CPU, No Errors, Rising Lag: The Kafka Consumer Is Waiting, Not Working

TL;DR: When kafka consumer lag rises while CPU remains low, it indicates the consumer is waiting on external I/O or is caught in an eviction loop rather than being under-provisioned. Before scaling out, you must identify if the lag is per-partition or group-wide to distinguish between a "poison pill" and a "rebalance storm."

You are on call. The alert fires: kafka consumer lag on your primary checkout service is climbing. You check the dashboard. The consumers are up, memory is stable at 40%, and CPU is idling at a cool 30%. There are no logged errors. The natural instinct is to scale the consumer group, but in many modern Kafka scenarios, adding more consumers will not only fail to fix the problem—it will actively make it worse.

This diagnostic guide moves past generic lists of causes to provide a branching decision procedure. By observing the "shape" of the lag, you can identify exactly why your consumer lag kafka is not decreasing.

What consumer lag actually measures (and what it does not)

To fix the issue, we must first define what we are measuring. In Kafka, lag is the difference between the Log-End Offset (LEO)—the last message written to the partition—and the Committed Offset.

It is critical to understand two distinctions that often trip up SREs during an incident:

1. Committed vs. Processed: A consumer might have physically read a message and even finished processing it, but if it hasn't successfully sent the offset commit back to the broker, the lag does not move.
2. Offset Lag vs. Time Lag: Offset lag counts messages. If your producers are idle, offset lag stays flat. Time lag (or "latency lag") measures the age of the oldest unprocessed message. According to WarpStream, offset lag cannot share one threshold across all workloads because 1,000 messages of lag might represent 1 second of work for a logging service but 1 hour of work for a heavy ML inference job.

Key Takeaway: Before scaling a lagging consumer group, look at lag per partition and its shape over time; each of five shapes has one confirming check, and two of them (eviction loop, poison pill) get worse if you add consumers.

Step 0: Read the lag per partition, not per group

A single aggregate number for a consumer group hides the truth. Your first step in any kafka consumer lag monitoring workflow should be to inspect the partitions individually.

Use the Kafka CLI to get the ground truth:

# Get the lag for every partition in the group
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group checkout-service-group

The output provides the state you need:

If you see one partition lagging while others are at zero, scaling the group will do nothing, as that partition is already owned by a single consumer.

The five shapes of lag and what they mean

1. All partitions rising together, steady slope: Throughput Deficit

If every partition shows a steady, synchronized climb in lag, you have a classic throughput deficit. However, if your CPU is low, the bottleneck isn't the code execution; it's I/O wait.

The Signature: Low CPU usage, high network wait, or high database lock contention. The Fix: Increase the parallelism of your downstream calls or optimize the external system (DB/API). If and only if the downstream system can handle the load, you can then add partitions and consumers.

2. Sawtooth that never recovers: The Poll-Interval Eviction Loop

This is the most common reason kafka consumer lag is not decreasing despite healthy-looking consumers. The consumer is taking too long to process a batch of records, causing the broker to think the consumer has died.

The Arithmetic of Failure: Consider the default Kafka settings:

• max.poll.records: 500
• max.poll.interval.ms: 300,000 (5 minutes)

If your per-record processing time (including DB writes) averages 700ms: 500 records * 700ms = 350,000ms (350 seconds)

Because 350s > 300s, the consumer is evicted from the group before it can commit the batch. A rebalance is triggered. The same batch is given to a new consumer, which also takes 350s and gets evicted. This is a rebalance storm.

Confirming Check: Check for the error message "Offset commit cannot be completed... consumer is not part of an active group." According to AWS MSK documentation, lag metrics may even disappear from your dashboard entirely while the group is in an UNSTABLE state during these storms. Why Scaling Fails: Adding consumers adds more overhead to the rebalance, extending the time the group spends doing nothing.

3. One partition climbing, the rest at zero: Poison Pill or Hot Key

If only one partition is lagging, you have a data-specific issue.

• Hot Key: One partition is receiving significantly more traffic because of a non-uniform hashing key (e.g., all "Guest" checkouts going to one partition).
• Poison Pill: A specific message at a specific offset is causing the consumer to crash or hang indefinitely.

Confirming Check: If the CURRENT-OFFSET for that partition is not moving at all, it’s a poison pill. If it is moving but slower than the LOG-END-OFFSET, it’s a hot key.

4. Lag flat and non-zero while producers are idle

Sometimes the lag stays at 500, even when no new messages are being produced.

• Stalled Commits: The consumer has processed the data but failed the final commit.
• Dead Consumer: The partition is assigned to a consumer that has hung (e.g., a deadlocked thread) but hasn't yet reached its session.timeout.ms.

Confirming Check: Switch to a time lag metric. If the offset lag is flat but time lag is increasing, the data is stale and the consumer is stuck.

5. Negative lag, or lag stuck at 1

You might see why kafka consumer lag is negative in your monitoring. This is usually a metric artifact rather than a real issue.

• Transactional Markers: In topics using Exactly-Once Semantics (EOS), control markers are written to the log but not included in the lag calculation until the transaction completes.
• Compaction: On compacted topics, the Log-End Offset might refer to a message that has been deleted, causing a temporary mismatch.
• Stale Metrics: Some exporters report -1 when they cannot fetch the offset.

A decision procedure for Kafka consumer lag

Alerting that matches the shapes

Basic threshold alerting on total group lag is insufficient. To capture these nuances, your kafka consumer lag monitoring should include:

1. Rebalance Rate: Alert if the group leaves the STABLE state more than twice in 10 minutes.
2. Time Lag (Latency): Alert if the oldest message in a partition exceeds a specific age (e.g., 5 minutes).
3. Missing Metric Alert: Especially on MSK, alert when the lag metric becomes "NaN" or disappears, as this indicates a rebalance storm.

Why this matters now

With the release of Apache Kafka 4.0 and the GA of KIP-848 (the new consumer group protocol), rebalances are becoming more efficient. As stated in the Apache Kafka 4.0 release announcement, the new protocol moves the complexity of rebalancing to the broker. However, the fundamental logic of processing timeouts still applies. Even with the CooperativeStickyAssignor, a consumer that fails to poll will eventually be evicted.

Common pitfalls

• Over-partitioning: Adding 100 partitions to solve a 10-partition bottleneck can lead to higher overhead and slower recovery during restarts.
• Ignoring Downstream Backpressure: If your database is the bottleneck, increasing Kafka throughput will only result in database connection pool exhaustion.
• Default Timeout Blindness: Many teams leave max.poll.interval.ms at 5 minutes, which is too low for heavy batch processing or high-latency third-party API calls.

Sources & further reading

• According to the Conduktor Glossary, monitoring time lag is often more descriptive of user impact than offset lag.
• According to Confluent Platform Documentation, broker-side lag metrics provide the most accurate view but are only available for groups using the official Kafka consumer API.
• According to the Apache Kafka Release Notes, KIP-848 significantly reduces the impact of rebalance storms but does not eliminate processing-time evictions.
• According to AWS MSK Documentation, CloudWatch metrics for lag are only emitted when the group is in a stable or empty state.

How Operate can help

Diagnosing these patterns manually while an incident is unfolding is stressful and error-prone. Operate is a self-hosted AI SRE platform that monitors your production environment to detect these specific lag signatures automatically. When a consumer group lag slope changes shape, Operate can correlate per-partition lag, rebalance events, and downstream database latency across your entire stack. It doesn't just alert you; it uses a Verification agent to test the hypothesis—identifying, for example, that a specific record is causing a processing timeout—and drafts a PR with a suggested max.poll.records adjustment or a code fix for the human on-call to review.

FAQ: Frequently Asked Questions

How to fix Kafka consumer lag?

First, identify the shape. If all partitions are lagging and CPU is high, scale out. If CPU is low, optimize downstream I/O. If the group is rebalancing constantly, increase your max.poll.interval.ms.

How is kafka consumer lag calculated?

It is the difference between the Log-End Offset (the highest offset in the partition) and the Committed Offset (the last offset the consumer successfully acknowledged to the broker).

Why is kafka consumer lag negative?

Negative lag is usually a metric artifact caused by transactional markers in Exactly-Once topics or stale offset metadata in the consumer group coordinator. It does not represent a real backlog.

Should I add partitions to reduce lag?

Only if your consumers are CPU-bound and you have already optimized your processing logic. If you are I/O bound or in a rebalance storm, adding partitions will not help.