Skip to content

Path 4 · lesson 5 of 5

Intermediate24 minApache Kafka 4.4 (tools, consumer metrics and configuration reference) · Spring Boot 3.5 with Spring for Apache Kafka 3.3, which manages kafka-clients 3.9

Consumer lag

What lag measures, how to read it per partition, how to tell a capacity problem from a stuck or hot partition, and what to change in each case.

What you will learn

  • Define lag for one partition of one group and explain why a group total hides the important cases
  • Read the output of kafka-consumer-groups.sh --describe column by column
  • Convert lag in records into lag in time and decide which one to alert on
  • Distinguish steady lag from growing lag and work out the consume rate a group can reach
  • Tell slow processing, too little parallelism, a stuck partition, rebalances and a hot partition apart from the lag pattern
  • Pick the fix that matches the cause instead of adding consumers by reflex

Before you start

The concept

For one consumer group and one partition, lag is the distance between the end of the log and the group's committed position:

Three consequences follow from the definition. Lag belongs to a group and a partition: two groups reading the same topic have unrelated lag, and a group has as many lag values as it has partitions. Lag is a difference of two moving numbers: it grows when producers get faster as well as when consumers get slower. And lag is measured against the committed offset, so a consumer that processes records but commits rarely looks further behind than it is.

Where you read itWhat it subtractsBlind spot
kafka-consumer-groups.sh --describe, or the Admin APILog end offset minus the group's committed offsetOverstates lag between commits. Works when every consumer is down.
Consumer metric records-lag and records-lag-maxLog end offset minus the consumer's current fetch positionReported by the consumer itself: a dead or stuck consumer reports nothing or a stale value.
The Kafka monitoring reference notes that records-lag-max is based on the current offset, not the committed offset.

Use both. The client metric is cheap and precise while the consumer is healthy. The committed-offset view from outside is the one that still works when the consumer is the thing that broke.

Lag is per partition

Simulation · not a real cluster

Watch the lag number on each partition. Crash a consumer and see which partitions fall behind while the others stay level.

Why it exists

In a request and response system, a slow server is felt at once: callers wait, time out and report errors. Kafka removes that coupling on purpose. A producer writes to the log and is done; it neither knows nor cares whether any consumer is keeping up. A consumer that runs at half the needed speed returns no errors, throws no exceptions and passes its health check. The only place where the problem exists is the growing gap between what has been written and what has been read. That gap is lag, which is why it is the first signal to put on a consumer's dashboard.

Lag is also what the business feels. A payment event that waits in the log for ten minutes is a customer looking at a pending order for ten minutes, whatever the consumer's CPU graph says.

Consumer lag & backpressure

Simulation, not a real cluster

An animated view of a consumer falling behind a producer and what backpressure does and does not do about it.

Step 1 · The pattern

A log between two speeds

The producer writes messages to a topic. A consumer reads each one and writes it to the database. The topic sits between them so neither has to wait for the other.

  • message
  • processed
Open the full animated lesson

How it works inside

Reading the tool

kafka-consumer-groups.sh --describeText
$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-serviceTOPIC          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG     CONSUMER-ID              HOST         CLIENT-IDorders.events  0          184210          184222          12      consumer-pay-1-5f0c...   /10.0.1.12   consumer-pay-1orders.events  1          183950          183958          8       consumer-pay-1-5f0c...   /10.0.1.12   consumer-pay-1orders.events  2          201377          248901          47524   consumer-pay-2-a91e...   /10.0.1.13   consumer-pay-2orders.events  3          184102          184117          15      consumer-pay-2-a91e...   /10.0.1.13   consumer-pay-2orders.events  4          90412           184066          93654   -                        -            -orders.events  5          183899          183903          4       consumer-pay-3-77bd...   /10.0.1.14   consumer-pay-3

Why it is written this way

  • CURRENT-OFFSET is the committed offset, LOG-END-OFFSET the end of the log, and LAG the difference. One row per partition the group has offsets for.
  • Partitions 0, 1, 3 and 5 show a few records of lag. That is normal: it is whatever arrived since the last commit.
  • Partition 2 has an owner and a large lag. Run the command again: if CURRENT-OFFSET is advancing, the partition is overloaded; if it is not moving at all, the consumer is stuck on a record.
  • Partition 4 has no CONSUMER-ID. Nobody owns it, so its lag grows at the full produce rate until a member is assigned.
  • Add --members --verbose to list members with their partitions, and --state for the group state and assignor.

Records and time

A lag of 50,000 records says little by itself. On a topic consumed at 5,000 records per second it is ten seconds of work. On a topic consumed at 10 per second it is almost an hour and a half. What users experience is time lag: how old the record being processed right now is. Kafka's built-in metrics report lag in records only, so time lag is something you derive, in one of two ways:

  • From the record. In the listener, subtract the record's timestamp from the current time. This is exact for the record in hand and costs nothing, but a consumer that has stopped reports no new values.
  • From the rates. Divide lag in records by the current consume rate to estimate the time to drain if producing stopped, or compare the timestamp of the record at the committed offset with the clock.

Alert on time lag against what the business can tolerate, and keep record lag for diagnosis. Record timestamps are set by the producer by default (message.timestamp.type is CreateTime), so a replay of old data shows up as a large record age even when the consumer is healthy.

Steady or growing

The rate of change of lag is the produce rate minus the consume rate. That gives three shapes worth recognising on a graph:

ShapeWhat it meansIs it a problem
Small and flatThe consumer keeps up. The residue is batching and the commit interval.No
Large and flatThe consumer matches the produce rate but never clears an old backlog. There is no spare capacity.Yes if the delay it represents matters, and the next traffic rise will tip it
GrowingRecords arrive faster than they are processed.Yes. It ends at the retention limit unless something changes
SawtoothConsumption stops and resumes: rebalances, long batches or pauses.Depends on the height of the teeth

To know whether a group can keep up at all, estimate its ceiling. A consumer that handles records of one partition one after another, at 10 ms each, processes at most 100 records per second for that partition. Six partitions with six such consumers reach about 600 per second in total, and no more. Extra consumers beyond the partition count are assigned nothing.

Produce rate against consume ceiling

Simulation · not a real cluster

The group starts below the produce rate. Change consumers, processing time and partitions and watch whether lag grows, holds or drains.

Causes, and how to tell them apart

CauseWhat the per-partition picture showsWhat to do
Slow processingAll partitions grow at a similar rate. Consumers spend nearly all their time in your code (poll-idle-ratio-avg near 0). A downstream latency graph usually rose first.Fix the slow dependency, batch the writes, or process in parallel within the consumer. More consumers help only if the dependency has headroom.
Too few consumers or partitionsAll partitions grow, processing time per record is normal, every consumer is busy.Add consumers up to the partition count. Beyond that, add partitions, knowing that keys then map to different partitions.
A stuck partitionOne partition's CURRENT-OFFSET does not move at all while its LOG-END-OFFSET climbs. Logs show the same offset failing repeatedly.Stop retrying the bad record forever: bounded retries, then a dead-letter topic. See the error handling lessons.
RebalancesSawtooth on many partitions at once, in step with deploys or with rebalance-rate-per-hour. Partitions briefly without a CONSUMER-ID.Cooperative assignor, static membership, batches that fit the poll interval. See the rebalancing lesson.
A hot partitionOne partition grows while its offset keeps advancing. Its LOG-END-OFFSET rises much faster than the others.Change the key or split the hot key's traffic. Adding consumers does nothing: one partition has one reader per group.

Architecture

Drawing diagram…

Flowchart. Starting from a lag alert, the first question is whether all partitions or only some are behind. All behind leads to checks on whether offsets advance, processing time and member count. A few behind leads to checks on ownership and whether the offset advances, ending at unassigned, stuck or hot partition.

flowchart TD
  A["Lag alert for a group"] --> B{"Which partitions are behind"}
  B -->|"All of them"| C{"Is the committed offset advancing"}
  B -->|"One or a few"| D{"Does the partition have an owner"}
  C -->|"No"| E["Group is down or in a rebalance loop"]
  C -->|"Yes"| F{"Is time per record higher than usual"}
  F -->|"Yes"| G["Slow processing or slow dependency"]
  F -->|"No"| H{"Fewer members than partitions"}
  H -->|"Yes"| I["Add consumers"]
  H -->|"No"| J["Add partitions or parallelise the work"]
  D -->|"No"| K["Unassigned, check members and session timeout"]
  D -->|"Yes"| L{"Is its committed offset advancing"}
  L -->|"No"| M["Stuck on a record, check errors"]
  L -->|"Yes"| N["Hot partition, check the key"]
A decision path from a lag alert to a cause, using only the per-partition view and two metrics.
  • The first split, all partitions or a few, separates capacity problems from partition problems. A group total cannot make this distinction, which is why lag must be collected per partition.
  • The second question on either branch is whether the committed offset is moving. Run the describe command twice a few seconds apart and compare CURRENT-OFFSET.
  • Only one leaf says to add consumers, and it is reached after ruling out slow processing and checking that the group has fewer members than partitions.
  • An unowned partition is a rebalancing or membership question, not a throughput one: look at the member list and at how long a missing member is allowed to hold its assignment.

Message flow

Drawing diagram…

Sequence diagram. A producer appends to a partition, a consumer fetches, processes and commits offset 1100, and a monitor reads the committed offset and the log end offset to get a lag of 100. The producer continues while the consumer is slow on its next batch, so the next measurement is 400.

sequenceDiagram
  participant P as Producer
  participant B as Broker partition 2
  participant C as Consumer
  participant G as Group coordinator
  participant M as Lag monitor
  P->>B: Append records, log end offset becomes 1200
  C->>B: Fetch from position 1000
  B-->>C: Records 1000 to 1099
  Note over C: Processes 100 records
  C->>G: Commit offset 1100
  M->>G: Read committed offset for the group
  G-->>M: 1100
  M->>B: Read log end offset
  B-->>M: 1200
  Note over M: Lag is 100 records
  P->>B: Append 300 more, log end offset becomes 1500
  C->>B: Fetch from position 1100
  B-->>C: Records 1100 to 1199
  Note over C: Slow database, this batch takes far longer
  M->>G: Read committed offset for the group
  G-->>M: Still 1100
  M->>B: Read log end offset
  B-->>M: 1500
  Note over M: Lag is 400 and growing
How an external monitor computes lag for one partition, and why slow processing shows up as growth.
  1. The monitor never talks to the consumer. It reads the committed offset from the group coordinator and the end offset from the partition leader, which is what the command-line tool does too.
  2. Between the two measurements the consumer has fetched records 1100 to 1199 but has not committed them. Its own records-lag metric would report 300 at this moment while the monitor reports 400: both are right about different things.
  3. Nothing in this exchange slows the producer down. The log end offset keeps rising at the produce rate whatever the consumer does.

In a real system

A typical inventory service consumes orders.events (12 partitions, keyed by order id) in the group inventory-service, running 4 instances with 3 listener threads each. For every event it reserves stock with one database transaction that takes about 8 ms, so each thread can handle roughly 125 events per second and the group about 1,500. Normal traffic is 400 events per second, and lag sits at a few dozen records per partition.

  • Sale day. Traffic rises to 2,500 per second. Every partition grows by about 80 records per second. The pattern is all partitions, offsets advancing, normal time per record, and members equal to partitions, so more instances would be idle. The options are more partitions, planned ahead because the key mapping changes, or batching several reservations into one transaction.
  • Slow database. A missing index takes the transaction from 8 ms to 60 ms. The ceiling falls to 200 per second, below normal traffic, and lag grows on all partitions. Doubling the instances would double the connections on a database that is already the bottleneck. The fix is in the query.
  • One bad event. An event references a product that does not exist and the handler throws every time. Partition 7 stops at one offset while the other eleven stay at zero lag. The fix is an error handler with bounded retries and a dead-letter topic.
  • A bulk customer. If the topic were keyed by customer id instead, a single wholesale customer could send a large share of all events to one partition. That partition would lag permanently while still making progress, and no number of consumers would change it.

In all four cases the group's total lag would show the same thing: a number going up. The per-partition view is what separates them.

Java and Spring Boot

Two measurements for a Spring Boot service: record age measured inside the listener, and committed-offset lag computed from outside the consumer with the Admin API, the same subtraction the command-line tool performs.

src/main/java/com/example/inventory/OrderEventsListener.javaJava
package com.example.inventory;import java.time.Duration;import io.micrometer.core.instrument.MeterRegistry;import io.micrometer.core.instrument.Timer;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderEventsListener {    private final StockReservations reservations;    private final Timer recordAge;    public OrderEventsListener(StockReservations reservations, MeterRegistry registry) {        this.reservations = reservations;        this.recordAge = Timer.builder("inventory.orders.record.age")                .description("Time between a record being produced and being picked up here")                .publishPercentiles(0.5, 0.99)                .register(registry);    }    @KafkaListener(topics = "orders.events", groupId = "inventory-service", concurrency = "3")    public void onOrderEvent(ConsumerRecord<String, OrderEvent> record) {        long ageMillis = Math.max(0, System.currentTimeMillis() - record.timestamp());        recordAge.record(Duration.ofMillis(ageMillis));        reservations.reserve(record.value());    }}

Why it is written this way

  • record.timestamp() is the producer's creation time by default, so the difference is the time lag of this record as the business experiences it.
  • The age is recorded before the work, so it measures waiting in the log and not the handler's own duration, which Spring already times as spring.kafka.listener.
  • Math.max(0, ...) guards against small clock differences between the producer host and this one.
  • This metric goes silent when the consumer stops. It must be paired with a measurement taken from outside, such as the next file.
  • OrderEvent is a record and StockReservations a service with an idempotent reserve method; both are assumed from the rest of the project.

Configuration

Lag has no setting of its own. These are the settings that change how fast a group consumes or how long a backlog survives, with defaults from the Apache Kafka 4.4 reference.

max.poll.recordsDefault 500
What it does
Upper limit on records handed to the application by one poll(). It does not change what is fetched from the broker.
When to change it
Raise it for batch listeners that write to a database in bulk. Lower it when single records are slow, so a batch stays inside max.poll.interval.ms.
fetch.min.bytesDefault 1
What it does
The broker answers a fetch as soon as this much data is available, or after fetch.max.wait.ms (default 500 ms).
When to change it
Raise it on high-volume topics to get fewer, fuller fetches. It adds up to fetch.max.wait.ms of latency on quiet topics.
max.partition.fetch.bytesDefault 1048576 (1 mebibyte)
What it does
Most data returned per partition in one fetch. fetch.max.bytes (50 mebibytes) limits the whole response.
When to change it
Raise it when draining a large backlog of small records with fast processing, where the fetch size is the limit. It is rarely the bottleneck when each record triggers a database call.
spring.kafka.listener.concurrencyDefault none (the container uses 1)
What it does
Number of consumer threads the listener container starts in this instance. Each is a separate group member.
When to change it
Raise it to use more partitions per instance. Threads across all instances beyond the partition count receive no partitions.
retention.ms (topic)Default 604800000 (7 days)
What it does
How long records are kept before they become eligible for deletion, whether or not any group has read them.
When to change it
It must comfortably exceed the longest outage and backlog you intend to recover from without losing data.
auto.offset.resetDefault latest
What it does
Where to start when the group has no committed offset for a partition, or the committed offset no longer exists in the log.
When to change it
With latest, a group that fell behind past retention silently skips to the end. Choose earliest or none when skipping is not acceptable.
offsets.retention.minutes (broker)Default 10080
What it does
How long committed offsets are kept after a group has become empty.
When to change it
A group stopped for longer than this loses its offsets and starts from auto.offset.reset when it returns.

Hands-on exercise

Find the ceiling, then find what scaling cannot fix

Make a group fall behind, bring it back by adding consumers, reach the limit set by the partition count, and then create two kinds of lag that more consumers do not cure.

Steps · 0 of 6 done

Lag under load

Simulation · not a real cluster

Change one control at a time and watch the consume rate, the per-partition lag and whether the total is rising or falling.

Take it further in the lab: Consumer lag during high load

What can go wrong

Alerting on total lag for the group
Symptom
A stuck partition goes unnoticed for hours because the group total stays under the threshold, or the alert fires on every traffic peak.
Cause
A sum mixes one partition that is not moving with many that are healthy, and a fixed record count means different delays at different traffic levels.
Fix
Alert on the worst partition, on lag that keeps growing over several minutes, and on time lag against a business limit.
Trusting only the consumer's own lag metric
Symptom
Dashboards show records-lag-max flat or absent during an outage while the command-line tool shows lag climbing.
Cause
The metric is published by the consumer and is based on its fetch position. A crashed consumer publishes nothing, and one that fetches but cannot commit looks caught up.
Fix
Also measure committed-offset lag from outside the consumer process, with the Admin API or an exporter built on it.
Adding consumers past the partition count
Symptom
The deployment is scaled from 6 to 12 instances and lag grows exactly as before. Half the members show no partitions.
Cause
A partition is read by one member of a group. Members beyond the partition count are spares.
Fix
Check members against partitions before scaling. If they are equal, the options are more partitions, faster processing or parallel work inside the consumer.
Clearing lag by resetting offsets to the end
Symptom
Lag drops to zero at once. Later, orders are found that were never processed.
Cause
Moving the committed offset to the log end does not process the backlog. It tells the group to skip it.
Fix
Drain the backlog with more capacity. Reset offsets only when skipping the data is an explicit decision, with the group stopped, using --reset-offsets with --dry-run first.
Adding partitions in the middle of an incident
Symptom
After the partition count is raised, events for the same order are processed out of order, and the old backlog is no smaller.
Cause
Existing records stay in their old partitions, so the backlog is not spread out. New records for a key may now go to a different partition than its earlier records.
Fix
Size partitions ahead of peaks. If you must add them, do it when the topic is drained or when per-key order across the change does not matter.

How to troubleshoot

Lag alert on a group

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group inventory-service
How to read it
Look at the LAG column per partition. All partitions behind is a capacity question. One or two behind is a partition question: check whether they have a CONSUMER-ID.
Need to know whether a lagging partition is slow or stuck

Check

Run the same describe command twice, about 10 seconds apart, and compare CURRENT-OFFSET
How to read it
Advancing but slower than LOG-END-OFFSET means overloaded or hot. Not advancing at all means stuck: look for the same offset failing in the logs, or for a consumer blocked in a call.
Unsure whether the consumers or the brokers are the limit

Check

Consumer metric `poll-idle-ratio-avg` (kafka.consumer:type=consumer-metrics) with `records-consumed-rate` (consumer-fetch-manager-metrics)
How to read it
An idle ratio near 0 means the thread is almost always in your code: processing is the limit. A high idle ratio with lag present means the consumer is waiting for data, so look at fetch settings, assignment or the broker.
Lag appears in spikes rather than a slope

Check

Consumer metrics `rebalance-rate-per-hour` and `last-rebalance-seconds-ago` (consumer-coordinator-metrics), and kafka-consumer-groups.sh --describe --group inventory-service --state
How to read it
Spikes that line up with rebalances, or a group state other than Stable, point at rebalancing rather than throughput.
Worried a backlog will reach retention

Check

Consumer metric `records-lead-min` (consumer-fetch-manager-metrics), or compare record age with the topic's retention.ms
How to read it
Lead is the distance from the consumer's position to the start of the log. A lead falling towards zero means unread records are about to be deleted.

Key takeaways

  • Lag is the log end offset minus the committed offset, for one group and one partition. Collect it per partition and look at the worst one.
  • The command-line tool measures against the committed offset from outside; the consumer's records-lag metrics measure against its fetch position from inside. Keep both.
  • Records measure the backlog, time measures the delay. Alert on time, diagnose with records.
  • Flat lag means the rates match; growing lag means the consume ceiling is below the produce rate, and retention is the deadline.
  • All partitions behind points at capacity or a slow dependency. One partition behind points at a stuck record, a hot key or a missing owner.
  • Adding consumers helps only below the partition count and only when processing itself is not the bottleneck.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A group of 8 members reads a 12 partition topic. Eleven partitions show lag under 50. Partition 6 shows a lag of 310,000, and two runs of the describe command one minute apart show its CURRENT-OFFSET rising by 18,000 while its LOG-END-OFFSET rises by 31,000. The other partitions each gain about 4,000 a minute.

What is the most likely cause and the fitting response?

Scenario 2 of 3

An order topic has 6 partitions and a group of 6 consumers. Each record takes 20 ms because of one database insert. Traffic has grown from 250 to 450 records per second and lag now rises on every partition at a similar rate. Database CPU is at 30 percent.

Which change brings lag back under control?

Scenario 3 of 3

During an incident a dashboard built on the consumer metric records-lag-max shows 0 for the payment group. At the same time kafka-consumer-groups.sh shows a lag of 40,000 on every partition, rising, with a CONSUMER-ID on each row. Application logs are full of commit failures.

How should you read the two numbers?

References

References: Apache Kafka 4.4: Basic operations (checking consumer position, managing consumer groups) ↗ · Apache Kafka 4.4: Monitoring (consumer fetch, coordinator and poll metrics) ↗ · Apache Kafka 4.4: Consumer configs ↗ · Apache Kafka 4.4: Topic configs (retention.ms, message.timestamp.type) ↗ · Kafka Admin Javadoc: listConsumerGroupOffsets and listOffsets ↗ · Spring for Apache Kafka 3.3: Monitoring (listener timers, Micrometer client metrics) ↗ · Spring for Apache Kafka 3.3: Listener container properties ↗ · Report a problem