Path 4 · lesson 5 of 5
Consumer lag
What lag measures, how to read it per partition, how to tell a capacity problem from a stuck or hot partition, and what to change in each case.
What you will learn
- Define lag for one partition of one group and explain why a group total hides the important cases
- Read the output of kafka-consumer-groups.sh --describe column by column
- Convert lag in records into lag in time and decide which one to alert on
- Distinguish steady lag from growing lag and work out the consume rate a group can reach
- Tell slow processing, too little parallelism, a stuck partition, rebalances and a hot partition apart from the lag pattern
- Pick the fix that matches the cause instead of adding consumers by reflex
Before you start
The concept
For one consumer group and one partition, lag is the distance between the end of the log and the group's committed position:
Three consequences follow from the definition. Lag belongs to a group and a partition: two groups reading the same topic have unrelated lag, and a group has as many lag values as it has partitions. Lag is a difference of two moving numbers: it grows when producers get faster as well as when consumers get slower. And lag is measured against the committed offset, so a consumer that processes records but commits rarely looks further behind than it is.
| Where you read it | What it subtracts | Blind spot |
|---|---|---|
kafka-consumer-groups.sh --describe, or the Admin API | Log end offset minus the group's committed offset | Overstates lag between commits. Works when every consumer is down. |
Consumer metric records-lag and records-lag-max | Log end offset minus the consumer's current fetch position | Reported by the consumer itself: a dead or stuck consumer reports nothing or a stale value. |
Use both. The client metric is cheap and precise while the consumer is healthy. The committed-offset view from outside is the one that still works when the consumer is the thing that broke.
Lag is per partition
Simulation · not a real clusterWatch the lag number on each partition. Crash a consumer and see which partitions fall behind while the others stay level.
Why it exists
In a request and response system, a slow server is felt at once: callers wait, time out and report errors. Kafka removes that coupling on purpose. A producer writes to the log and is done; it neither knows nor cares whether any consumer is keeping up. A consumer that runs at half the needed speed returns no errors, throws no exceptions and passes its health check. The only place where the problem exists is the growing gap between what has been written and what has been read. That gap is lag, which is why it is the first signal to put on a consumer's dashboard.
Lag is also what the business feels. A payment event that waits in the log for ten minutes is a customer looking at a pending order for ten minutes, whatever the consumer's CPU graph says.
Consumer lag & backpressure
Simulation, not a real clusterAn animated view of a consumer falling behind a producer and what backpressure does and does not do about it.
Step 1 · The pattern
A log between two speeds
The producer writes messages to a topic. A consumer reads each one and writes it to the database. The topic sits between them so neither has to wait for the other.
- message
- processed
How it works inside
Reading the tool
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-serviceTOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-IDorders.events 0 184210 184222 12 consumer-pay-1-5f0c... /10.0.1.12 consumer-pay-1orders.events 1 183950 183958 8 consumer-pay-1-5f0c... /10.0.1.12 consumer-pay-1orders.events 2 201377 248901 47524 consumer-pay-2-a91e... /10.0.1.13 consumer-pay-2orders.events 3 184102 184117 15 consumer-pay-2-a91e... /10.0.1.13 consumer-pay-2orders.events 4 90412 184066 93654 - - -orders.events 5 183899 183903 4 consumer-pay-3-77bd... /10.0.1.14 consumer-pay-3Why it is written this way
CURRENT-OFFSETis the committed offset,LOG-END-OFFSETthe end of the log, andLAGthe difference. One row per partition the group has offsets for.- Partitions 0, 1, 3 and 5 show a few records of lag. That is normal: it is whatever arrived since the last commit.
- Partition 2 has an owner and a large lag. Run the command again: if
CURRENT-OFFSETis advancing, the partition is overloaded; if it is not moving at all, the consumer is stuck on a record. - Partition 4 has no
CONSUMER-ID. Nobody owns it, so its lag grows at the full produce rate until a member is assigned. - Add
--members --verboseto list members with their partitions, and--statefor the group state and assignor.
Records and time
A lag of 50,000 records says little by itself. On a topic consumed at 5,000 records per second it is ten seconds of work. On a topic consumed at 10 per second it is almost an hour and a half. What users experience is time lag: how old the record being processed right now is. Kafka's built-in metrics report lag in records only, so time lag is something you derive, in one of two ways:
- From the record. In the listener, subtract the record's timestamp from the current time. This is exact for the record in hand and costs nothing, but a consumer that has stopped reports no new values.
- From the rates. Divide lag in records by the current consume rate to estimate the time to drain if producing stopped, or compare the timestamp of the record at the committed offset with the clock.
Alert on time lag against what the business can tolerate, and keep record lag for diagnosis. Record timestamps are set by the producer by default (message.timestamp.type is CreateTime), so a replay of old data shows up as a large record age even when the consumer is healthy.
Steady or growing
The rate of change of lag is the produce rate minus the consume rate. That gives three shapes worth recognising on a graph:
| Shape | What it means | Is it a problem |
|---|---|---|
| Small and flat | The consumer keeps up. The residue is batching and the commit interval. | No |
| Large and flat | The consumer matches the produce rate but never clears an old backlog. There is no spare capacity. | Yes if the delay it represents matters, and the next traffic rise will tip it |
| Growing | Records arrive faster than they are processed. | Yes. It ends at the retention limit unless something changes |
| Sawtooth | Consumption stops and resumes: rebalances, long batches or pauses. | Depends on the height of the teeth |
To know whether a group can keep up at all, estimate its ceiling. A consumer that handles records of one partition one after another, at 10 ms each, processes at most 100 records per second for that partition. Six partitions with six such consumers reach about 600 per second in total, and no more. Extra consumers beyond the partition count are assigned nothing.
Produce rate against consume ceiling
Simulation · not a real clusterThe group starts below the produce rate. Change consumers, processing time and partitions and watch whether lag grows, holds or drains.
Causes, and how to tell them apart
| Cause | What the per-partition picture shows | What to do |
|---|---|---|
| Slow processing | All partitions grow at a similar rate. Consumers spend nearly all their time in your code (poll-idle-ratio-avg near 0). A downstream latency graph usually rose first. | Fix the slow dependency, batch the writes, or process in parallel within the consumer. More consumers help only if the dependency has headroom. |
| Too few consumers or partitions | All partitions grow, processing time per record is normal, every consumer is busy. | Add consumers up to the partition count. Beyond that, add partitions, knowing that keys then map to different partitions. |
| A stuck partition | One partition's CURRENT-OFFSET does not move at all while its LOG-END-OFFSET climbs. Logs show the same offset failing repeatedly. | Stop retrying the bad record forever: bounded retries, then a dead-letter topic. See the error handling lessons. |
| Rebalances | Sawtooth on many partitions at once, in step with deploys or with rebalance-rate-per-hour. Partitions briefly without a CONSUMER-ID. | Cooperative assignor, static membership, batches that fit the poll interval. See the rebalancing lesson. |
| A hot partition | One partition grows while its offset keeps advancing. Its LOG-END-OFFSET rises much faster than the others. | Change the key or split the hot key's traffic. Adding consumers does nothing: one partition has one reader per group. |
Architecture
Drawing diagram…
Flowchart. Starting from a lag alert, the first question is whether all partitions or only some are behind. All behind leads to checks on whether offsets advance, processing time and member count. A few behind leads to checks on ownership and whether the offset advances, ending at unassigned, stuck or hot partition.
flowchart TD
A["Lag alert for a group"] --> B{"Which partitions are behind"}
B -->|"All of them"| C{"Is the committed offset advancing"}
B -->|"One or a few"| D{"Does the partition have an owner"}
C -->|"No"| E["Group is down or in a rebalance loop"]
C -->|"Yes"| F{"Is time per record higher than usual"}
F -->|"Yes"| G["Slow processing or slow dependency"]
F -->|"No"| H{"Fewer members than partitions"}
H -->|"Yes"| I["Add consumers"]
H -->|"No"| J["Add partitions or parallelise the work"]
D -->|"No"| K["Unassigned, check members and session timeout"]
D -->|"Yes"| L{"Is its committed offset advancing"}
L -->|"No"| M["Stuck on a record, check errors"]
L -->|"Yes"| N["Hot partition, check the key"]- The first split, all partitions or a few, separates capacity problems from partition problems. A group total cannot make this distinction, which is why lag must be collected per partition.
- The second question on either branch is whether the committed offset is moving. Run the describe command twice a few seconds apart and compare
CURRENT-OFFSET. - Only one leaf says to add consumers, and it is reached after ruling out slow processing and checking that the group has fewer members than partitions.
- An unowned partition is a rebalancing or membership question, not a throughput one: look at the member list and at how long a missing member is allowed to hold its assignment.
Message flow
Drawing diagram…
Sequence diagram. A producer appends to a partition, a consumer fetches, processes and commits offset 1100, and a monitor reads the committed offset and the log end offset to get a lag of 100. The producer continues while the consumer is slow on its next batch, so the next measurement is 400.
sequenceDiagram participant P as Producer participant B as Broker partition 2 participant C as Consumer participant G as Group coordinator participant M as Lag monitor P->>B: Append records, log end offset becomes 1200 C->>B: Fetch from position 1000 B-->>C: Records 1000 to 1099 Note over C: Processes 100 records C->>G: Commit offset 1100 M->>G: Read committed offset for the group G-->>M: 1100 M->>B: Read log end offset B-->>M: 1200 Note over M: Lag is 100 records P->>B: Append 300 more, log end offset becomes 1500 C->>B: Fetch from position 1100 B-->>C: Records 1100 to 1199 Note over C: Slow database, this batch takes far longer M->>G: Read committed offset for the group G-->>M: Still 1100 M->>B: Read log end offset B-->>M: 1500 Note over M: Lag is 400 and growing
- The monitor never talks to the consumer. It reads the committed offset from the group coordinator and the end offset from the partition leader, which is what the command-line tool does too.
- Between the two measurements the consumer has fetched records 1100 to 1199 but has not committed them. Its own
records-lagmetric would report 300 at this moment while the monitor reports 400: both are right about different things. - Nothing in this exchange slows the producer down. The log end offset keeps rising at the produce rate whatever the consumer does.
In a real system
A typical inventory service consumes orders.events (12 partitions, keyed by order id) in the group inventory-service, running 4 instances with 3 listener threads each. For every event it reserves stock with one database transaction that takes about 8 ms, so each thread can handle roughly 125 events per second and the group about 1,500. Normal traffic is 400 events per second, and lag sits at a few dozen records per partition.
- Sale day. Traffic rises to 2,500 per second. Every partition grows by about 80 records per second. The pattern is all partitions, offsets advancing, normal time per record, and members equal to partitions, so more instances would be idle. The options are more partitions, planned ahead because the key mapping changes, or batching several reservations into one transaction.
- Slow database. A missing index takes the transaction from 8 ms to 60 ms. The ceiling falls to 200 per second, below normal traffic, and lag grows on all partitions. Doubling the instances would double the connections on a database that is already the bottleneck. The fix is in the query.
- One bad event. An event references a product that does not exist and the handler throws every time. Partition 7 stops at one offset while the other eleven stay at zero lag. The fix is an error handler with bounded retries and a dead-letter topic.
- A bulk customer. If the topic were keyed by customer id instead, a single wholesale customer could send a large share of all events to one partition. That partition would lag permanently while still making progress, and no number of consumers would change it.
In all four cases the group's total lag would show the same thing: a number going up. The per-partition view is what separates them.
Java and Spring Boot
Two measurements for a Spring Boot service: record age measured inside the listener, and committed-offset lag computed from outside the consumer with the Admin API, the same subtraction the command-line tool performs.
package com.example.inventory;import java.time.Duration;import io.micrometer.core.instrument.MeterRegistry;import io.micrometer.core.instrument.Timer;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderEventsListener { private final StockReservations reservations; private final Timer recordAge; public OrderEventsListener(StockReservations reservations, MeterRegistry registry) { this.reservations = reservations; this.recordAge = Timer.builder("inventory.orders.record.age") .description("Time between a record being produced and being picked up here") .publishPercentiles(0.5, 0.99) .register(registry); } @KafkaListener(topics = "orders.events", groupId = "inventory-service", concurrency = "3") public void onOrderEvent(ConsumerRecord<String, OrderEvent> record) { long ageMillis = Math.max(0, System.currentTimeMillis() - record.timestamp()); recordAge.record(Duration.ofMillis(ageMillis)); reservations.reserve(record.value()); }}Why it is written this way
record.timestamp()is the producer's creation time by default, so the difference is the time lag of this record as the business experiences it.- The age is recorded before the work, so it measures waiting in the log and not the handler's own duration, which Spring already times as
spring.kafka.listener. Math.max(0, ...)guards against small clock differences between the producer host and this one.- This metric goes silent when the consumer stops. It must be paired with a measurement taken from outside, such as the next file.
OrderEventis a record andStockReservationsa service with an idempotentreservemethod; both are assumed from the rest of the project.
package com.example.inventory;import java.util.HashMap;import java.util.Map;import java.util.TreeMap;import java.util.concurrent.TimeUnit;import jakarta.annotation.PreDestroy;import org.apache.kafka.clients.admin.Admin;import org.apache.kafka.clients.admin.ListOffsetsResult.ListOffsetsResultInfo;import org.apache.kafka.clients.admin.OffsetSpec;import org.apache.kafka.clients.consumer.OffsetAndMetadata;import org.apache.kafka.common.TopicPartition;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.core.KafkaAdmin;import org.springframework.scheduling.annotation.Scheduled;import org.springframework.stereotype.Component;/** Computes log end offset minus committed offset per partition, like the CLI does. */@Componentpublic class GroupLagReporter { private static final Logger log = LoggerFactory.getLogger(GroupLagReporter.class); private static final String GROUP = "inventory-service"; private final Admin admin; public GroupLagReporter(KafkaAdmin kafkaAdmin) { this.admin = Admin.create(kafkaAdmin.getConfigurationProperties()); } @Scheduled(fixedDelay = 30_000) public void report() throws Exception { Map<String, Long> lagByPartition = currentLag(); long worst = lagByPartition.values().stream().mapToLong(Long::longValue).max().orElse(0); log.info("group={} worstPartitionLag={} perPartition={}", GROUP, worst, lagByPartition); } Map<String, Long> currentLag() throws Exception { Map<TopicPartition, OffsetAndMetadata> committed = admin .listConsumerGroupOffsets(GROUP) .partitionsToOffsetAndMetadata() .get(10, TimeUnit.SECONDS); Map<TopicPartition, OffsetSpec> latest = new HashMap<>(); committed.keySet().forEach(partition -> latest.put(partition, OffsetSpec.latest())); Map<TopicPartition, ListOffsetsResultInfo> ends = admin .listOffsets(latest) .all() .get(10, TimeUnit.SECONDS); Map<String, Long> lag = new TreeMap<>(); committed.forEach((partition, offset) -> { if (offset != null) { long end = ends.get(partition).offset(); lag.put(partition.toString(), Math.max(0, end - offset.offset())); } }); return lag; } @PreDestroy public void close() { admin.close(); }}Why it is written this way
- It reports the worst partition, not the sum. A sum hides one stuck partition behind eleven healthy ones.
- The Admin client reads committed offsets from the coordinator, so the result is correct even when no consumer in the group is running. In production this job belongs in a separate process or a lag exporter, so that it survives the consumer it watches.
- Only partitions with a committed offset appear. A partition the group has never committed for is missing from the map, not reported as zero.
- The two reads are not atomic, so the value can be off by whatever was produced or committed in between.
Math.max(0, ...)hides the occasional negative result. @Scheduledneeds@EnableSchedulingon a configuration class. Spring Boot provides theKafkaAdminbean.
Configuration
Lag has no setting of its own. These are the settings that change how fast a group consumes or how long a backlog survives, with defaults from the Apache Kafka 4.4 reference.
max.poll.recordsDefault 500
- What it does
- Upper limit on records handed to the application by one
poll(). It does not change what is fetched from the broker. - When to change it
- Raise it for batch listeners that write to a database in bulk. Lower it when single records are slow, so a batch stays inside
max.poll.interval.ms.
fetch.min.bytesDefault 1
- What it does
- The broker answers a fetch as soon as this much data is available, or after
fetch.max.wait.ms(default 500 ms). - When to change it
- Raise it on high-volume topics to get fewer, fuller fetches. It adds up to
fetch.max.wait.msof latency on quiet topics.
max.partition.fetch.bytesDefault 1048576 (1 mebibyte)
- What it does
- Most data returned per partition in one fetch.
fetch.max.bytes(50 mebibytes) limits the whole response. - When to change it
- Raise it when draining a large backlog of small records with fast processing, where the fetch size is the limit. It is rarely the bottleneck when each record triggers a database call.
spring.kafka.listener.concurrencyDefault none (the container uses 1)
- What it does
- Number of consumer threads the listener container starts in this instance. Each is a separate group member.
- When to change it
- Raise it to use more partitions per instance. Threads across all instances beyond the partition count receive no partitions.
retention.ms (topic)Default 604800000 (7 days)
- What it does
- How long records are kept before they become eligible for deletion, whether or not any group has read them.
- When to change it
- It must comfortably exceed the longest outage and backlog you intend to recover from without losing data.
auto.offset.resetDefault latest
- What it does
- Where to start when the group has no committed offset for a partition, or the committed offset no longer exists in the log.
- When to change it
- With
latest, a group that fell behind past retention silently skips to the end. Chooseearliestornonewhen skipping is not acceptable.
offsets.retention.minutes (broker)Default 10080
- What it does
- How long committed offsets are kept after a group has become empty.
- When to change it
- A group stopped for longer than this loses its offsets and starts from
auto.offset.resetwhen it returns.
Hands-on exercise
Find the ceiling, then find what scaling cannot fix
Make a group fall behind, bring it back by adding consumers, reach the limit set by the partition count, and then create two kinds of lag that more consumers do not cure.
Steps · 0 of 6 done
Lag under load
Simulation · not a real clusterChange one control at a time and watch the consume rate, the per-partition lag and whether the total is rising or falling.
What you should see
Two consumers at 10 ms per record manage about 200 per second, so lag grows by about 200 per second. Four consumers match the produce rate: lag stops growing but the backlog stays. Six give spare capacity and the backlog drains; nine consume no faster than six because there are only six partitions. At 40 ms per record six consumers reach about 150 per second and adding consumers cannot help. With a hot key, one partition falls behind while the rest stay near zero. With a pool of 2, consumers queue for connections and the ceiling drops whatever the consumer count.
What can go wrong
Alerting on total lag for the group
- Symptom
- A stuck partition goes unnoticed for hours because the group total stays under the threshold, or the alert fires on every traffic peak.
- Cause
- A sum mixes one partition that is not moving with many that are healthy, and a fixed record count means different delays at different traffic levels.
- Fix
- Alert on the worst partition, on lag that keeps growing over several minutes, and on time lag against a business limit.
Trusting only the consumer's own lag metric
- Symptom
- Dashboards show
records-lag-maxflat or absent during an outage while the command-line tool shows lag climbing. - Cause
- The metric is published by the consumer and is based on its fetch position. A crashed consumer publishes nothing, and one that fetches but cannot commit looks caught up.
- Fix
- Also measure committed-offset lag from outside the consumer process, with the Admin API or an exporter built on it.
Adding consumers past the partition count
- Symptom
- The deployment is scaled from 6 to 12 instances and lag grows exactly as before. Half the members show no partitions.
- Cause
- A partition is read by one member of a group. Members beyond the partition count are spares.
- Fix
- Check members against partitions before scaling. If they are equal, the options are more partitions, faster processing or parallel work inside the consumer.
Clearing lag by resetting offsets to the end
- Symptom
- Lag drops to zero at once. Later, orders are found that were never processed.
- Cause
- Moving the committed offset to the log end does not process the backlog. It tells the group to skip it.
- Fix
- Drain the backlog with more capacity. Reset offsets only when skipping the data is an explicit decision, with the group stopped, using
--reset-offsetswith--dry-runfirst.
Adding partitions in the middle of an incident
- Symptom
- After the partition count is raised, events for the same order are processed out of order, and the old backlog is no smaller.
- Cause
- Existing records stay in their old partitions, so the backlog is not spread out. New records for a key may now go to a different partition than its earlier records.
- Fix
- Size partitions ahead of peaks. If you must add them, do it when the topic is drained or when per-key order across the change does not matter.
How to troubleshoot
Lag alert on a group
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group inventory-service
- How to read it
- Look at the
LAGcolumn per partition. All partitions behind is a capacity question. One or two behind is a partition question: check whether they have aCONSUMER-ID.
Need to know whether a lagging partition is slow or stuck
Check
Run the same describe command twice, about 10 seconds apart, and compare CURRENT-OFFSET
- How to read it
- Advancing but slower than
LOG-END-OFFSETmeans overloaded or hot. Not advancing at all means stuck: look for the same offset failing in the logs, or for a consumer blocked in a call.
Unsure whether the consumers or the brokers are the limit
Check
Consumer metric `poll-idle-ratio-avg` (kafka.consumer:type=consumer-metrics) with `records-consumed-rate` (consumer-fetch-manager-metrics)
- How to read it
- An idle ratio near 0 means the thread is almost always in your code: processing is the limit. A high idle ratio with lag present means the consumer is waiting for data, so look at fetch settings, assignment or the broker.
Lag appears in spikes rather than a slope
Check
Consumer metrics `rebalance-rate-per-hour` and `last-rebalance-seconds-ago` (consumer-coordinator-metrics), and kafka-consumer-groups.sh --describe --group inventory-service --state
- How to read it
- Spikes that line up with rebalances, or a group state other than
Stable, point at rebalancing rather than throughput.
Worried a backlog will reach retention
Check
Consumer metric `records-lead-min` (consumer-fetch-manager-metrics), or compare record age with the topic's retention.ms
- How to read it
- Lead is the distance from the consumer's position to the start of the log. A lead falling towards zero means unread records are about to be deleted.
Key takeaways
- Lag is the log end offset minus the committed offset, for one group and one partition. Collect it per partition and look at the worst one.
- The command-line tool measures against the committed offset from outside; the consumer's
records-lagmetrics measure against its fetch position from inside. Keep both. - Records measure the backlog, time measures the delay. Alert on time, diagnose with records.
- Flat lag means the rates match; growing lag means the consume ceiling is below the produce rate, and retention is the deadline.
- All partitions behind points at capacity or a slow dependency. One partition behind points at a stuck record, a hot key or a missing owner.
- Adding consumers helps only below the partition count and only when processing itself is not the bottleneck.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A group of 8 members reads a 12 partition topic. Eleven partitions show lag under 50. Partition 6 shows a lag of 310,000, and two runs of the describe command one minute apart show its CURRENT-OFFSET rising by 18,000 while its LOG-END-OFFSET rises by 31,000. The other partitions each gain about 4,000 a minute.
What is the most likely cause and the fitting response?
Scenario 2 of 3
An order topic has 6 partitions and a group of 6 consumers. Each record takes 20 ms because of one database insert. Traffic has grown from 250 to 450 records per second and lag now rises on every partition at a similar rate. Database CPU is at 30 percent.
Which change brings lag back under control?
Scenario 3 of 3
During an incident a dashboard built on the consumer metric records-lag-max shows 0 for the payment group. At the same time kafka-consumer-groups.sh shows a lag of 40,000 on every partition, rising, with a CONSUMER-ID on each row. Application logs are full of commit failures.
How should you read the two numbers?
References
References: Apache Kafka 4.4: Basic operations (checking consumer position, managing consumer groups) ↗ · Apache Kafka 4.4: Monitoring (consumer fetch, coordinator and poll metrics) ↗ · Apache Kafka 4.4: Consumer configs ↗ · Apache Kafka 4.4: Topic configs (retention.ms, message.timestamp.type) ↗ · Kafka Admin Javadoc: listConsumerGroupOffsets and listOffsets ↗ · Spring for Apache Kafka 3.3: Monitoring (listener timers, Micrometer client metrics) ↗ · Spring for Apache Kafka 3.3: Listener container properties ↗ · Report a problem