Skip to content

Path 4 · lesson 1 of 5

Foundation22 minApache Kafka 4.x (defaults checked against the 4.0 and 4.4 docs) · Java 21 · Spring Boot 3.4 · Spring for Apache Kafka 3.3 (kafka-clients 3.8 to 3.9)

The consumer and its poll loop

A Kafka consumer pulls records by calling poll in a loop. See what one poll does, why the time between polls matters, and how Spring wraps the loop.

What you will learn

  • Describe what a single call to poll does: fetching, group membership, rebalance callbacks and auto-commit
  • Write a correct plain-Java poll loop with a clean shutdown using wakeup
  • Work out whether a batch can be processed inside max.poll.interval.ms and choose max.poll.records accordingly
  • Tell apart the two liveness checks: heartbeats against the session timeout, and polling against the max poll interval
  • Explain what Spring's listener container does around your @KafkaListener method

Before you start

The concept

A Kafka broker never sends a record to a consumer on its own initiative. The consumer asks. Your application calls poll(Duration) on a KafkaConsumer, gets back a batch of records (possibly empty), processes them, and calls poll again. That loop is the whole consumer. Everything else a consumer does, such as joining its group, finding out which partitions it owns, running rebalance callbacks and committing offsets automatically, is driven from inside that same call.

The consumer keeps a position for every partition it reads: the offset of the next record poll will hand out. The position moves forward each time poll returns records. It lives in the consumer's memory only. The durable version, the committed offset, is what a restarted consumer resumes from, and it is the subject of Offset management.

PieceWhat it isWho drives it
FetchA request to a partition leader for records after the current position, up to the high watermarkSent while you are inside poll
Record bufferFetched batches held in memory. poll hands them out at most max.poll.records at a timeFilled by fetch responses, drained by poll
Group membershipJoining the group, receiving an assignment, giving partitions upCompleted inside poll
HeartbeatA periodic signal to the group coordinator that the process is aliveA background thread, independent of your processing
Auto-commitPeriodic commit of the positions reached, when enable.auto.commit=trueTriggered from poll, and on close
The parts of a consumer and what moves each one.

Why it exists

The alternative is a broker that pushes. A pushing broker has to decide how fast each consumer can take data, and it has to guess. When it guesses too high the consumer is flooded, and the broker then needs per-consumer flow control, buffering and redelivery state. Kafka's design notes give this as the first reason for pulling: the consumer sets the pace, so a slow consumer simply falls behind in the log and catches up later, instead of being overwhelmed.

Pulling also makes batching natural. A consumer that is behind asks once and receives everything available from its position up to a size limit, in one response. A push system has to choose between sending each record immediately and holding records back without knowing whether the receiver is ready.

The usual objection to pulling is busy-waiting: if there is nothing to read, the consumer spins. Kafka avoids that with a long poll. The fetch request tells the broker to hold the request until at least fetch.min.bytes of data exists or fetch.max.wait.ms has passed. An idle consumer therefore costs roughly one request per half second per broker it reads from, and a new record is still delivered almost as soon as it is readable.

Broker pushes

  • Broker chooses the rate and must track each consumer's capacity
  • A slow consumer needs broker-side buffering or dropped deliveries
  • Batch size is a guess made by the sender
  • Replaying old data needs a special mechanism

Consumer pulls (Kafka)

  • Consumer chooses the rate. Falling behind shows up as lag, not as overload
  • The broker keeps no per-consumer delivery state beyond committed offsets
  • A consumer that is behind receives large batches automatically
  • Replay is only a matter of moving the position with seek

How it works inside

What one call to poll does

  1. Group work. If the consumer used subscribe, the first poll finds the group coordinator and joins the group. Later polls take part in any rebalance. Your ConsumerRebalanceListener callbacks (onPartitionsRevoked, onPartitionsAssigned) run here, on your thread, which is why poll can block for longer than the timeout you passed.
  2. Auto-commit. With enable.auto.commit=true, poll checks whether auto.commit.interval.ms has elapsed and, if so, commits the positions reached so far.
  3. Position lookup. For a partition with no position yet, the consumer fetches the group's committed offset. If there is none, auto.offset.reset decides where to start.
  4. Return buffered records. If earlier fetches left records in the buffer, poll returns up to max.poll.records of them at once and advances the positions.
  5. Fetch. Otherwise it sends fetch requests to the leaders of its assigned partitions and waits up to the timeout. The consumer also sends the next fetches before returning, so data is usually already waiting when you call poll again.

Fetch sizing

A fetch covers many partitions on one broker. The broker answers as soon as fetch.min.bytes (default 1 byte) is available, or after fetch.max.wait.ms (default 500 ms) if it is not. Each partition contributes at most max.partition.fetch.bytes (default 1 MiB) and the whole response at most fetch.max.bytes (default 50 MiB). Neither is a hard cap: if the first batch in the first non-empty partition is larger than the limit it is returned anyway, so a consumer can never get stuck behind one oversized batch. A consumer is only ever given records below the partition's high watermark, that is, records already replicated to the in-sync replicas.

Two liveness checks, two failure modes

With the classic group protocol the consumer runs a heartbeat thread in the background. It tells the coordinator the process is alive every heartbeat.interval.ms (default 3 seconds). If the coordinator hears nothing for session.timeout.ms (default 45 seconds), it removes the member and rebalances. This catches a crashed process or a broken network.

A process can be alive and still stuck: a deadlock, an endless retry, a database call with no timeout. The heartbeat thread would keep the member in the group for ever while its partitions make no progress. So there is a second check on the processing thread. If poll is not called again within max.poll.interval.ms (default 5 minutes), the client itself leaves the group so another member can take the partitions. A later commitSync then fails with CommitFailedException, because only active members may commit.

CheckDetectsSetting and defaultRuns on
HeartbeatDead process, lost networksession.timeout.ms 45 s, heartbeat.interval.ms 3 sBackground heartbeat thread
Poll intervalLive process that stopped making progressmax.poll.interval.ms 5 minMeasured between your calls to poll

This is why processing time between polls matters. The budget is simple arithmetic: records per poll multiplied by the worst-case time per record must stay well under max.poll.interval.ms. At the defaults, 500 records at 700 ms each is 350 seconds, which is over the 300 second limit. The consumer is thrown out of the group mid-batch, the partitions move, and the new owner reprocesses everything since the last commit. What happens next is covered in Consumer rebalancing.

How Spring wraps the loop

Spring for Apache Kafka runs the loop for you. A KafkaMessageListenerContainer owns one KafkaConsumer and one thread. That thread calls poll with the container's pollTimeout (default 5 seconds), then calls your @KafkaListener method once per record (or once per batch for a batch listener), then commits according to the container's AckMode, and polls again. A ConcurrentMessageListenerContainer with concurrency = 3 is three of those containers: three consumers, three threads.

Two consequences follow. Your listener method runs on the consumer's own thread, so a slow listener is exactly a slow gap between polls and is subject to max.poll.interval.ms. And since version 2.3 the container sets enable.auto.commit to false unless you set it yourself, because it commits offsets itself after the listener returns.

Architecture

Drawing diagram…

Flowchart of a payment service process containing a poll thread, a record buffer and a heartbeat thread. The poll thread sends fetch requests to two partition leaders, whose responses fill the buffer, sends group and commit requests to the group coordinator, and writes to a payments database. The heartbeat thread talks only to the group coordinator.

flowchart LR
  subgraph APP["Payment service process"]
    PT["Poll thread: poll, process, commit"]
    BUF["Record buffer"]
    HB["Heartbeat thread"]
    PT -->|"takes up to max.poll.records"| BUF
  end
  subgraph CLUSTER["Kafka cluster"]
    L0["Broker 1: leader of orders.events-0"]
    L1["Broker 2: leader of orders.events-1"]
    GC["Broker 3: group coordinator"]
  end
  L0 -->|"fetch response"| BUF
  L1 -->|"fetch response"| BUF
  PT -->|"fetch requests"| L0
  PT -->|"fetch requests"| L1
  PT -->|"join group, commit offsets"| GC
  HB -->|"heartbeat every 3 s"| GC
  PT -->|"one record at a time"| DB["Payments database"]
One consumer: a poll thread that fetches, processes and commits, and a heartbeat thread that only proves the process is alive.
  • Fetches go to the brokers that lead the assigned partitions. Group membership and offset commits go to a different broker role, the group coordinator, which is one specific broker for each group.
  • The record buffer decouples network batches from application batches: one fetch response can feed several polls.
  • The heartbeat thread never touches your records. It can keep reporting a healthy member while the poll thread is stuck in the database call, which is exactly why the poll interval check exists.
  • Everything on the poll thread is sequential. Time spent in the database is time not spent polling.

Message flow

Drawing diagram…

Sequence diagram. The poll thread calls poll, the consumer joins the group through the coordinator, receives its assignment and committed offsets, fetches 780 records from the partition leader and returns 500 of them. The application processes them and commits offset 620. The second poll returns the remaining 280 records from the buffer without a network round trip, and the next fetch is held by the broker until data arrives.

sequenceDiagram
  participant A as Poll thread
  participant C as KafkaConsumer
  participant L as Partition leader
  participant G as Group coordinator
  A->>C: poll with 1 s timeout
  C->>G: join group and sync assignment
  G-->>C: assigned orders.events 0 and 1
  C->>G: fetch committed offsets
  G-->>C: partition 0 at 120, partition 1 at 98
  C->>L: fetch from 120, min 1 byte, wait up to 500 ms
  L-->>C: records 120 to 899
  C-->>A: 500 records, 120 to 619
  Note over C,G: heartbeat thread keeps sending heartbeats
  A->>A: process 500 records
  A->>C: commitSync
  C->>G: commit partition 0 offset 620
  A->>C: poll again
  C-->>A: 280 buffered records, 620 to 899
  C->>L: next fetch from 900
  L-->>C: held until data arrives or 500 ms pass
The first two polls of a consumer: join, find the starting offsets, fetch, hand out the buffer in slices, and long-poll when caught up.
  1. The join and the committed-offset lookup happen inside the first poll, not in subscribe. subscribe only records what you want.
  2. The leader returned 780 records but poll returned 500. The other 280 were already in memory, so the second poll returned immediately.
  3. The commit sent offset 620, the next record to read, after records 120 to 619 were processed.
  4. Once the consumer is caught up, the broker holds each fetch for up to fetch.max.wait.ms. That is the long poll that prevents busy-waiting.

In a real system

Take a typical payment service that reacts to orders. The Order Service publishes OrderPlaced events to the topic orders.events, which has 6 partitions and is keyed by orderId so that all events of one order stay in order. The Payment Service runs 3 instances in the consumer group payment-service, so each instance reads 2 partitions. For each event it authorises the payment with an external provider and writes a row to its database, which takes about 40 ms normally and up to 2 seconds when the provider is slow.

SituationPer recordBatch of 500Against the 5 minute limit
Normal day40 ms20 sComfortable
Provider degraded2 s1,000 sExceeded: the member leaves the group mid-batch
Provider degraded, max.poll.records=502 s100 sInside the limit, with room to spare
The same consumer under a slow dependency. The budget is records per poll times worst-case time per record.

The team therefore sets max.poll.records to 50 and puts a 3 second timeout on the provider call. The cost is nothing on the network side, because the fetch size is unchanged. The gain is that a bad day shows up as growing lag, which is visible and recoverable, instead of a loop of rebalances and reprocessed payments. The full system is in the example project.

Java and Spring Boot

First the loop written by hand against the Kafka client, so that nothing is hidden. Then the same consumer as a Spring Boot listener, where the container runs the loop. Both read orders.events in the group payment-service.

OrderEventPollLoop.javaJava
package com.example.payments.consumer;import java.time.Duration;import java.util.List;import java.util.Properties;import java.util.concurrent.atomic.AtomicBoolean;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.consumer.ConsumerRecords;import org.apache.kafka.clients.consumer.KafkaConsumer;import org.apache.kafka.common.errors.WakeupException;import org.apache.kafka.common.serialization.StringDeserializer;/** A hand-written poll loop. Run it on its own thread, one instance per consumer. */public class OrderEventPollLoop implements Runnable {    private final AtomicBoolean closed = new AtomicBoolean(false);    private final KafkaConsumer<String, String> consumer;    public OrderEventPollLoop(String bootstrapServers) {        Properties props = new Properties();        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);        props.put(ConsumerConfig.GROUP_ID_CONFIG, "payment-service");        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");        props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "50");        this.consumer = new KafkaConsumer<>(props);    }    @Override    public void run() {        try {            consumer.subscribe(List.of("orders.events"));            while (!closed.get()) {                ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));                for (ConsumerRecord<String, String> record : records) {                    handle(record);                }                if (!records.isEmpty()) {                    consumer.commitSync();                }            }        } catch (WakeupException e) {            if (!closed.get()) {                throw e;            }        } finally {            consumer.close();        }    }    /** Safe to call from another thread, for example a JVM shutdown hook. */    public void shutdown() {        closed.set(true);        consumer.wakeup();    }    private void handle(ConsumerRecord<String, String> record) {        System.out.printf("partition=%d offset=%d key=%s%n",                record.partition(), record.offset(), record.key());        // authorise the payment and write the result; must finish well inside max.poll.interval.ms    }}

Why it is written this way

  • subscribe only registers interest. Joining the group, receiving the assignment and looking up the committed offsets all happen inside the first poll.
  • poll(Duration.ofSeconds(1)) blocks for at most about a second when there is nothing to read, so the closed flag is checked regularly even on a quiet topic.
  • Auto-commit is off and commitSync() runs after the whole batch is handled. A crash in the middle of a batch means the batch is read again: at-least-once.
  • max.poll.records=50 bounds the work between two polls. It does not make the network fetches smaller.
  • wakeup() is the only consumer method that may be called from another thread. It makes a blocked poll throw WakeupException, and the finally block then closes the consumer, which leaves the group promptly instead of waiting for the session timeout.

Configuration

Consumer client settings that shape the poll loop, with the defaults documented for Kafka 4.x. The same defaults apply to the 3.8 and 3.9 clients that Spring Boot 3.4 brings in. In Spring Boot, set them with the dedicated spring.kafka.consumer.* property where one exists, otherwise under spring.kafka.consumer.properties.*.

max.poll.recordsDefault 500
What it does
The most records one call to poll returns. It does not change how much is fetched from the broker.
When to change it
Lower it when per-record processing is slow or unpredictable, so that a batch always fits inside max.poll.interval.ms. Raise it only for cheap in-memory work.
max.poll.interval.msDefault 300000 (5 minutes)
What it does
The longest allowed gap between two calls to poll. Past it, the consumer is considered failed and leaves the group.
When to change it
Raise it for genuinely long batch jobs, knowing that a stuck consumer is then detected later and rebalances can take longer. Prefer a smaller batch first.
fetch.min.bytesDefault 1
What it does
The least data the broker should have before answering a fetch. With 1 byte the broker answers as soon as anything is available.
When to change it
Raise it (for example to tens of kilobytes) on high-volume topics to get fewer, larger responses, accepting up to fetch.max.wait.ms of extra latency.
fetch.max.wait.msDefault 500
What it does
How long the broker may hold a fetch that cannot yet satisfy fetch.min.bytes.
When to change it
This is the ceiling on the latency that fetch.min.bytes can add. Lower it if you raised fetch.min.bytes and need a tighter bound.
max.partition.fetch.bytesDefault 1048576 (1 mebibyte)
What it does
The most data returned per partition in one fetch. A first batch larger than this is still returned so the consumer can progress.
When to change it
Raise it when records are large and you want more than a few per partition in each fetch. Memory use grows with the number of assigned partitions.
fetch.max.bytesDefault 52428800 (50 mebibytes)
What it does
The most data returned for one fetch request across all its partitions. Also not an absolute maximum.
When to change it
Lower it on memory-constrained consumers that read many partitions. The consumer sends several fetches in parallel, one per broker.
session.timeout.msDefault 45000 (45 seconds)
What it does
Classic protocol: how long the coordinator waits without a heartbeat before removing the member.
When to change it
Lower it for faster detection of dead instances, at the risk of false alarms during long garbage collection pauses. Not supported with group.protocol=consumer, where the broker setting group.consumer.session.timeout.ms applies.
heartbeat.interval.msDefault 3000 (3 seconds)
What it does
Classic protocol: the expected time between heartbeats to the coordinator.
When to change it
Keep it at no more than a third of session.timeout.ms. Not supported with group.protocol=consumer, where the broker setting group.consumer.heartbeat.interval.ms (default 5 seconds) applies.
spring.kafka.listener.poll-timeoutDefault 5s (container default)
What it does
The timeout the Spring listener container passes to poll.
When to change it
Rarely needs changing. A shorter value makes the container notice pause, resume and stop requests sooner on an idle topic.

Hands-on exercise

Find the batch size that survives a slow dependency

Use the throughput simulation to see how processing time per record and max.poll.records decide whether a consumer keeps up, and work out the poll interval budget by hand.

Steps · 0 of 5 done

Processing time between polls

Simulation · not a real cluster

Slow the per-record processing down and watch what happens to the batch time and the lag. Then change the batch size.

What can go wrong

A batch that takes longer than the poll interval
Symptom
Frequent rebalances, the same records processed again and again, and CommitFailedException from commitSync. In Spring the log says the member was removed because the time between polls exceeded max.poll.interval.ms.
Cause
Records per poll times time per record went over max.poll.interval.ms, usually when a downstream dependency slowed down. The client left the group while the batch was still running.
Fix
Lower max.poll.records, put timeouts on every external call, and only then consider raising max.poll.interval.ms. Fix the slow dependency rather than hiding it.
Sharing one consumer between threads
Symptom
ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access.
Cause
A second thread called a consumer method, for example a scheduled task calling commitSync or a worker thread calling pause, while the poll thread was using the consumer.
Fix
Keep every consumer call on the thread that polls. Use one consumer per thread, and let other threads signal through a queue or flag. Only wakeup() is safe from outside.
Blocking for ever inside the listener
Symptom
One consumer stops making progress on its partitions, is removed after five minutes, and the next owner gets stuck on the same record.
Cause
An HTTP or database call without a timeout, or an unbounded in-method retry loop, holds the poll thread. The heartbeat thread keeps the member alive until the poll interval expires.
Fix
Set connect and read timeouts on every client. Use the container's error handler with a bounded back-off for retries instead of sleeping in the listener.
Stopping the process without closing the consumer
Symptom
After a deploy, the partitions of the stopped instance are not read for about 45 seconds.
Cause
The process was killed without close(), so the coordinator only notices through the missing heartbeats after session.timeout.ms.
Fix
Shut down cleanly: wakeup() then close() in a plain consumer. In Spring, let the application context stop the containers and give the pod enough termination grace time.

How to troubleshoot

Records are produced but the consumer logs nothing

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
Compare CURRENT-OFFSET with LOG-END-OFFSET. If they are equal the group has already read everything, or it started at the end of the log because auto.offset.reset is latest. If LAG is growing and CONSUMER-ID is empty, no member owns that partition.
The group rebalances every few minutes under load

Check

Consumer log: 'consumer poll timeout has expired' followed by the member leaving the group
How to read it
The time between two polls went over max.poll.interval.ms. Measure the batch time (records per poll times time per record) and reduce max.poll.records.
Throughput is low even though the consumer is not busy

Check

Consumer metrics `records-per-request-avg` and `fetch-latency-avg` (group consumer-fetch-manager-metrics)
How to read it
Few records per request with fetch latency near 500 ms means the consumer is caught up and long-polling, so the producer side is the limit. Many records per request with a busy poll thread means processing is the limit.
Not sure whether an instance is still in the group

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --members --verbose
How to read it
Each active member is listed with its client id, host and assigned partitions. An instance that is running but missing from the list has left the group, most often through the poll interval.

Key takeaways

  • Consumers pull. The consumer sets its own pace, and a slow consumer shows up as lag instead of being overloaded.
  • poll does much more than fetch: group membership, rebalance callbacks and auto-commit all happen inside it.
  • max.poll.records limits what one poll returns, not what is fetched. Lowering it is a cheap way to shorten the gap between polls.
  • Heartbeats prove the process is alive. Calling poll proves it is making progress. Each has its own timeout.
  • Records per poll times worst-case time per record must stay well below max.poll.interval.ms.
  • One consumer, one thread. Spring's listener container is that thread, and your listener method runs on it.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A Spring Boot listener reads invoices and renders a PDF for each one. Rendering normally takes 150 ms. After a template change it takes 900 ms. The consumer uses default settings. The group now rebalances about every five minutes and the same invoices are rendered repeatedly.

Which change addresses the cause with the least side effect?

Scenario 2 of 3

A team wants more throughput from a plain-Java consumer. They keep one KafkaConsumer, and after each poll they hand the records to a pool of 8 worker threads. Each worker calls consumer.commitSync() when it finishes its record.

What will they see first?

Scenario 3 of 3

A consumer on a low-traffic topic receives about one record per second. Someone sets fetch.min.bytes to 1 MiB to make it more efficient and leaves fetch.max.wait.ms at its default. Records are about 1 KiB each.

How does delivery change?

References

References: Apache Kafka 4.4 documentation: Consumer Configs ↗ · KafkaConsumer Javadoc (Kafka 4.4): detecting consumer failures, multi-threaded processing ↗ · Apache Kafka 4.4 documentation: Design, push vs. pull ↗ · Apache Kafka 4.4 documentation: Consumer Rebalance Protocol (KIP-848 status and settings) ↗ · KIP-62: Allow consumer to send heartbeats from a background thread ↗ · Spring for Apache Kafka 3.3 reference: Message Listener Containers ↗ · Spring for Apache Kafka 3.3 reference: @KafkaListener Annotation ↗ · Report a problem