Path 9 · lesson 2 of 6
@KafkaListener in depth: threads, acknowledgements, batches
What runs behind @KafkaListener: the listener container and its threads, when offsets are committed in each ack mode, batch listeners, concurrency against partitions, and pausing.
What you will learn
- Describe the container, consumer and thread that each @KafkaListener creates, and what runs on that thread
- Choose listener method parameters for the job: payload, ConsumerRecord, headers, Acknowledgment or a batch
- Say exactly when offsets are committed in each AckMode and what a crash at each point redelivers
- Set concurrency from the partition count and the number of instances, and predict which consumers sit idle
- Keep a listener inside max.poll.interval.ms, and judge when handing work to another thread is safe
- Pause, resume, stop and start a listener at runtime through KafkaListenerEndpointRegistry
Before you start
The concept
@KafkaListener on a method looks like a callback, but it declares a small runtime. For each annotated method Spring creates a listener container. The container owns one or more Kafka consumers, runs the poll loop for each on its own thread, converts every record to your method's parameters, calls the method, and commits offsets afterwards according to its acknowledgement mode.
The single most useful fact about it: the thread that calls your method is the thread that polls. While your method runs, that consumer is not polling. Everything about speed, rebalances, pausing and commit timing follows from that.
| Attribute | What it sets | Notes |
|---|---|---|
topics / topicPattern | The subscription; Kafka's group management assigns partitions | Placeholders such as ${app.topics.orders-placed} and SpEL are allowed |
topicPartitions | Explicit partitions with optional start offsets, without group assignment | For special cases such as reading a whole compacted topic on every instance |
groupId | The consumer group for this listener, overriding spring.kafka.consumer.group-id | Each group receives every record once |
id | The container's id, used to find it in the registry and in thread names | Also becomes the group id when groupId is not set, unless idIsGroup = false |
concurrency | Number of consumers and threads for this listener in this instance | A string, so it can be a placeholder; overrides the factory's value |
containerFactory | Bean name of the container factory to use | Default is kafkaListenerContainerFactory |
autoStartup | Whether the container starts with the application | "false" for listeners you start through the registry |
batch | Deliver each poll as a list instead of record by record | Overrides the factory's batch setting |
properties | Kafka consumer properties for this listener only, as key=value strings | group.id and client.id are ignored here; use groupId and clientIdPrefix |
clientIdPrefix | Prefix of the consumers' client.id | A suffix -n is added per consumer thread |
errorHandler, filter, containerPostProcessor | Bean names of per-listener extensions | Container-level error handling is the subject of the next but one lesson |
Why it exists
A hand-written consumer is a loop: poll, process each record, commit, repeat, on a thread you manage. It is short, and it has to get several things right at once: commit only what was processed, never touch the consumer from a second thread (KafkaConsumer is not thread-safe), keep calling poll often enough to stay in the group, handle a failing record without ending the loop, and stop cleanly on shutdown.
The listener container is that loop written once. It gives you a declarative method and takes over threading, commits, error handling hooks and lifecycle. The price is that the loop is now out of sight. People who do not know it is there make the same mistakes as before, only less visibly: slow work inside the listener, work handed to another thread while the container commits, or more threads than partitions.
How it works inside
Containers, consumers and threads
Boot's factory creates a ConcurrentMessageListenerContainer per listener. It holds concurrency child containers (KafkaMessageListenerContainer), and each child owns one KafkaConsumer and one thread. The thread name contains the listener id and the child index, for example payment-orders-0-C-1, which makes log lines attributable to a consumer.
- The consumer thread calls
pollwith the container'spollTimeout(5 seconds by default) and receives up tomax.poll.recordsrecords (client default 500) from its assigned partitions. - For a record listener it calls your method once per record, in order, on the same thread. For a batch listener it calls the method once with the whole list.
- After the listener returns, the container decides whether to commit, based on the ack mode.
- It polls again. Only now can it receive new records, learn about a rebalance, or act on a pause or stop request.
With the classic group protocol, heartbeats come from a separate background thread of the Kafka client, so a slow listener does not trip session.timeout.ms. What it trips is max.poll.interval.ms (default 5 minutes): if the time between two poll calls exceeds it, the consumer leaves the group, its partitions are given to others, and its next commit fails. The records it was working on are delivered again to the new owner.
One listener object serves all consumer threads. With concurrency = 3 your method runs on three threads at once, each with records from different partitions. Keep listeners stateless or thread-safe.
What the method can receive
| Parameter | What you get | Use it when |
|---|---|---|
The payload type, e.g. OrderPlaced (optionally @Payload) | The deserialised and converted record value | The value is all you need |
ConsumerRecord<K, V> | The raw record: key, value, topic, partition, offset, timestamp, headers | You log or store the record's coordinates, or need the headers |
@Header(KafkaHeaders.RECEIVED_KEY) and similar | One piece of metadata: RECEIVED_KEY, RECEIVED_TOPIC, RECEIVED_PARTITION, OFFSET, RECEIVED_TIMESTAMP | You want the payload type plus one or two coordinates |
ConsumerRecordMetadata | Everything from the record except key and value | You want all coordinates without the raw record |
Acknowledgment | A handle to acknowledge (or nack) this record or batch | The ack mode is MANUAL or MANUAL_IMMEDIATE |
Consumer<?, ?> | The underlying Kafka consumer | Rarely: only on the listener thread, and never to seek or commit behind the container's back |
List<OrderPlaced>, List<ConsumerRecord<K, V>> or ConsumerRecords<K, V> | The whole poll (batch listeners) | The work is cheaper in bulk, such as a JDBC batch insert |
RECEIVED_KEY is absent when the record key is null, so mark that header required = false if keys can be missing.For batch listeners the reference is strict: a List<ConsumerRecord<K, V>> or ConsumerRecords<K, V> parameter must be the only parameter apart from an optional Acknowledgment and Consumer. Annotations such as @Payload and @Header must be on the concrete method, not on an interface it implements.
Ack modes: when the commit happens
Since version 2.3 the container sets enable.auto.commit to false unless you set it yourself, because it has its own commit logic. The Kafka client's default is true, so this is a place where Spring behaves differently from a plain consumer. The container then commits according to its AckMode; commits are synchronous by default (syncCommits).
| AckMode | Commit happens | Redelivered after a crash |
|---|---|---|
BATCH (default) | When all records returned by the poll have been processed | Every record of the current poll, including those already processed |
RECORD | After the listener returns from each record | At most the record in progress |
TIME / COUNT / COUNT_TIME | After a poll's records are processed, once ackTime has passed or ackCount records were received | Everything since the last commit: possibly several polls |
MANUAL | The listener calls acknowledge(); the commit is then done as in BATCH, after the poll's records | Like BATCH, limited to what was acknowledged |
MANUAL_IMMEDIATE | At once when acknowledge() is called on the listener thread | Whatever was not acknowledged |
RECORD narrows the duplicate window to one record for the cost of one commit request per record. BATCH is the cheapest and repeats up to a whole poll. Neither removes the need for an idempotent consumer: a crash between your database commit and the offset commit repeats the record in every mode.
The manual modes exist for cases where returning from the method is not the right moment: you acknowledge only after a downstream call confirms, or decline with nack(Duration), which commits what was acknowledged before, discards the rest of the poll and redelivers from the failed record after the given sleep. nack must be called on the listener thread, and it pauses the whole listener, not one partition. If your method takes an Acknowledgment but the container is not in a manual mode, the reference says the parameter will be null; the container factory decides the mode, not the method signature.
When the listener throws, the container's error handler takes over. By default that is a DefaultErrorHandler, which redelivers the failed record a limited number of times and then logs and skips it. Configuring it, and sending failures to a dead letter topic, is the subject of Spring error handling.
Concurrency against partitions
concurrency is the number of consumers one application instance contributes to the group for this listener. The group's total is instances multiplied by concurrency, and Kafka assigns each partition to exactly one of them. Consumers beyond the partition count receive nothing.
Five listener threads, three partitions
Simulation · not a real clusterThink of this as one instance with concurrency = 5 on a topic with three partitions. See which consumers get a partition and which stay idle, then remove consumers one by one.
- Six partitions, two instances,
concurrency = 3: six consumers, one partition each. Adding a third instance adds three idle consumers, which is useful only as standby. - Six partitions, one instance,
concurrency = 2: two threads with three partitions each. Records of different partitions are interleaved on a thread, but each partition is still processed in order. - A listener on several topics with the 3.9 client's default assignor list uses
RangeAssignor, which assigns per topic. The reference's example: three topics of five partitions withconcurrency = 15leaves ten consumers idle, because each topic's five partitions go to the same first five consumers.RoundRobinAssignororCooperativeStickyAssignorspreads them.
More threads than partitions do not add throughput. If one partition's records are slow, the options are more partitions (with the key-ordering consequences from partition sizing), faster processing, or bulk work in a batch listener.
Handing work to another thread
The tempting fix for a slow listener is to submit the work to an executor and return immediately. Know what you give up:
- The delivery guarantee. In
BATCHorRECORDmode the container commits when the method returns. The work is still in an in-memory queue. A crash loses it: at-most-once, without anyone having chosen that. - Ordering. Two records of the same partition can now run at the same time or finish in the opposite order.
- Backpressure. The consumer keeps polling at full speed while the executor queue grows until memory runs out or tasks are rejected.
If you must process asynchronously, keep the commit tied to completion. Spring for Apache Kafka offers two supported ways: the container property asyncAcks with a manual ack mode, where acknowledgements may arrive in any order and the container defers commits until the gaps are filled (the consumer is paused until all offsets of the previous poll are committed); and, since 3.2, listener methods that return CompletableFuture<?> or Mono<?>, for which the container switches to manual acknowledgement with out-of-order commits and acknowledges when the future completes. The reference notes that both increase the chance of duplicate deliveries after a failure, and neither restores per-key ordering. In most services the better answer is to stay synchronous, lower max.poll.records, and scale with partitions.
Pause, resume, stop, start
Containers created for @KafkaListener are not beans. They are registered in the KafkaListenerEndpointRegistry, where getListenerContainer(id) returns one by the annotation's id.
pause() and resume()
- The consumer stays in the group and keeps its partitions
- The container keeps calling
poll, so no rebalance happens, but no records are returned - Takes effect before the next poll: records of the current poll are still delivered, unless
pauseImmediateis set - Lag grows on this consumer's partitions; nobody else picks them up
- For short dependency outages and backpressure
stop() and start()
- The consumers close and leave the group
- A rebalance gives their partitions to the remaining members, if there are any
- By default the records of the current poll are processed first;
stopImmediatestops after the current record start()rejoins and triggers another rebalance- For maintenance, feature switches and listeners with
autoStartup = "false"
Architecture
Drawing diagram…
A flowchart. A KafkaListener method with id payment-orders and concurrency 3 is registered in the KafkaListenerEndpointRegistry, and the container factory builds a ConcurrentMessageListenerContainer for it. That container has three child containers, each with one thread and one consumer, named payment-orders-0-C-1 to payment-orders-2-C-1. The topic orders.placed has six partitions, two assigned to each child. All three threads call the same PaymentListener bean, after which the container commits offsets for the group payment-service according to the ack mode.
flowchart TB
Ann["KafkaListener method, id payment-orders, concurrency 3"] --> Reg["KafkaListenerEndpointRegistry"]
Fac["kafkaListenerContainerFactory"] --> CMLC
Reg --> CMLC["ConcurrentMessageListenerContainer payment-orders"]
subgraph Children["Three child containers, each one thread and one consumer"]
C0["payment-orders-0-C-1"]
C1["payment-orders-1-C-1"]
C2["payment-orders-2-C-1"]
end
CMLC --> C0
CMLC --> C1
CMLC --> C2
subgraph Topic["orders.placed, 6 partitions"]
P01["Partitions 0 and 1"]
P23["Partitions 2 and 3"]
P45["Partitions 4 and 5"]
end
P01 --> C0
P23 --> C1
P45 --> C2
C0 --> Bean["One PaymentListener bean, called from all three threads"]
C1 --> Bean
C2 --> Bean
Bean --> Ack["Container commits offsets per AckMode"]
Ack --> Off["Group payment-service offsets"]- The annotation is only a declaration. The registry holds the container, and the
idis how you reach it at runtime. - Each child container is an independent group member with its own consumer, poll loop and assigned partitions. A slow record on one thread delays only that thread's partitions.
- There is one listener bean. Instance fields are shared by all threads.
- Commits are made by the same thread that polled and processed, after the listener returns. Nothing commits in the background.
Message flow
Drawing diagram…
A sequence diagram. The consumer thread polls and receives offsets 40 and 41 of orders.placed partition 0. It calls the listener for offset 40, the listener inserts a payment and returns, and the thread commits offset 41. It then calls the listener for offset 41. The database is slow and the insert takes six minutes, during which the thread does not poll. The max poll interval passes and partition 0 is reassigned. When the listener returns, the commit of offset 42 fails because the member no longer owns the partition, and offset 41 is delivered again to the new owner.
sequenceDiagram participant B as Broker and group coordinator participant T as Consumer thread payment-orders-0-C-1 participant L as PaymentListener participant D as Payments database T->>B: poll B-->>T: orders.placed-0 offsets 40 and 41 T->>L: onOrderPlaced offset 40 L->>D: insert payment for order o-40 D-->>L: committed L-->>T: return T->>B: commit offset 41, AckMode RECORD T->>L: onOrderPlaced offset 41 L->>D: insert payment for order o-41 Note over L,D: slow dependency, the thread is not polling D-->>L: committed after 6 minutes L-->>T: return Note over B,T: max.poll.interval.ms passed, partition 0 reassigned T->>B: commit offset 42 B-->>T: commit fails, member no longer owns the partition B-->>T: offset 41 is delivered again to the new owner
- In
RECORDmode the commit after offset 40 is for offset 41: the position of the next record to read. - While the listener waits for the database, nothing else happens on this thread. No poll, no commit, no reaction to a pause or stop request.
- After
max.poll.interval.msthe group gives the partition to another member. The payment for o-41 was already written, the commit fails, and the new owner processes o-41 again. - The second processing must be harmless. That is the job of an idempotent consumer, not of any listener setting.
In a real system
The payment service of the example project reads orders.placed (6 partitions, key = order id) in the group payment-service. It runs two instances with concurrency = 3, so each of the six consumers owns one partition. Charging a card is a remote call that usually takes 150 ms and occasionally several seconds.
| Decision | Value | Reason |
|---|---|---|
| Ack mode | RECORD | A redelivered poll of 50 charge requests is expensive to deduplicate; one record is not |
max.poll.records | 50 | 50 records at a worst case of 3 s each is 150 s, inside the 300 s interval |
max.poll.interval.ms | 300000 (default) | Left alone: raising it also delays the detection of a stuck consumer |
| Concurrency | 3 per instance, 2 instances | Equals the six partitions; a third instance would be standby only |
| Provider outage | Pause the container | Stops consumption without leaving the group; lag grows visibly and drains after resume |
| Duplicates | Unique constraint on order id in the payments table | Redelivery after a crash or rebalance cannot charge twice |
A second listener in the same service writes every order event to a reporting table. Nobody waits for it and the cost is the database round trip, so it is a batch listener in its own group payment-reporting: one multi-row insert per poll, and one commit per poll.
Partitions & consumer groups
Simulation, not a real clusterThe animated Build lesson shows partitions being shared within a group and what an extra consumer does.
Step 1 · The pattern
A log and a reader
The producer appends messages to a partition: an ordered log. A consumer reads them one after another, in the order they were written.
- message
- consumed
Java and Spring Boot
Three listeners of the payment service and the code around them: a record listener with metadata, a manual-acknowledgement variant, a batch listener, the extra container factory, and runtime control through the registry. Written against the Spring for Apache Kafka 3.3 reference; not compiled in this environment.
spring: kafka: consumer: group-id: payment-service auto-offset-reset: earliest max-poll-records: 50 key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer properties: "[spring.deserializer.value.delegate.class]": org.springframework.kafka.support.serializer.JsonDeserializer "[spring.json.value.default.type]": com.example.payments.events.OrderPlaced "[spring.json.use.type.headers]": false "[partition.assignment.strategy]": org.apache.kafka.clients.consumer.CooperativeStickyAssignor listener: ack-mode: record concurrency: 3 missing-topics-fatal: trueapp: topics: orders-placed: orders.placedWhy it is written this way
spring.kafka.listener.*configures Boot's default container factory.ack-mode: recordandconcurrency: 3apply to every listener that does not override them.enable.auto.commitis not mentioned. The container sets it to false and commits itself. Setting it to true here would add timer-based commits that ignore whether your listener has finished.max-poll-records: 50is chosen from the processing time, as in the table above. It limits how much work must fit intomax.poll.interval.msand how many records aBATCHmode crash repeats.- The deserializer lines are explained in Spring serialisation.
package com.example.payments.events;import java.time.Instant;/** The payment service's own view of the event published to orders.placed. */public record OrderPlaced( String eventId, String orderId, String customerId, long amountMinor, String currency, Instant occurredAt) {}Why it is written this way
- The consumer has its own record in its own package. It shares the JSON field names with the producer, not a Java class.
package com.example.payments.kafka;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.support.KafkaHeaders;import org.springframework.messaging.handler.annotation.Header;import org.springframework.messaging.handler.annotation.Payload;import org.springframework.stereotype.Component;import com.example.payments.events.OrderPlaced;import com.example.payments.service.PaymentService;@Componentpublic class PaymentListener { public static final String LISTENER_ID = "payment-orders"; private static final Logger log = LoggerFactory.getLogger(PaymentListener.class); private final PaymentService paymentService; public PaymentListener(PaymentService paymentService) { this.paymentService = paymentService; } @KafkaListener( id = LISTENER_ID, groupId = "payment-service", topics = "${app.topics.orders-placed}", concurrency = "${app.payment-listener.concurrency:3}") public void onOrderPlaced( @Payload OrderPlaced event, @Header(KafkaHeaders.RECEIVED_PARTITION) int partition, @Header(KafkaHeaders.OFFSET) long offset) { log.info("Charging order {} from {}-{}", event.orderId(), partition, offset); // Synchronous on the consumer thread. Idempotent on orderId: a redelivered // record finds the existing payment and does nothing. paymentService.chargeOnce(event); }}Why it is written this way
idnames the container for the registry and for thread names.groupIdis set explicitly; without it theidwould become the consumer group.- Topic and concurrency are placeholders.
concurrencyis a string attribute for exactly this reason; the default after the colon applies when the property is absent. - The payload type drives conversion. Partition and offset are injected as headers and logged with the order id, so any record can be found again with a console consumer.
- The method does the work on the consumer thread and returns only when the charge is recorded. The container then commits (
RECORDmode fromapplication.yml). If it throws, nothing is committed for this record and the container's error handler decides what happens. chargeOncemust be idempotent. A crash after the database commit and before the offset commit redelivers the record, in every ack mode. The pattern is in the idempotent consumer.- The bean is called from three threads at once. It has no mutable fields.
package com.example.payments.config;import org.springframework.boot.autoconfigure.kafka.ConcurrentKafkaListenerContainerFactoryConfigurer;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;import org.springframework.kafka.core.ConsumerFactory;import org.springframework.kafka.listener.ContainerProperties;@Configurationpublic class KafkaListenerConfig { @Bean ConcurrentKafkaListenerContainerFactory<Object, Object> manualAckKafkaListenerContainerFactory( ConcurrentKafkaListenerContainerFactoryConfigurer configurer, ConsumerFactory<Object, Object> consumerFactory) { ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>(); configurer.configure(factory, consumerFactory); factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE); return factory; }}Why it is written this way
- The ack mode belongs to the container, so a listener that wants a different one needs a different factory. This one is additional: Boot's
kafkaListenerContainerFactorystill exists. configurer.configure(...)first applies everything fromspring.kafka.listener.*and Boot's error handler and converter beans. Only the ack mode is changed afterwards.- Container properties are set through
getContainerProperties()on the factory; the factory copies them into each container it creates. - If every listener in the service should acknowledge manually, no bean is needed: set
spring.kafka.listener.ack-mode: manual_immediate.
package com.example.payments.kafka;import java.time.Duration;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.support.Acknowledgment;import org.springframework.stereotype.Component;import com.example.payments.events.RefundRequested;import com.example.payments.service.ProviderUnavailableException;import com.example.payments.service.RefundService;@Componentpublic class RefundListener { private final RefundService refundService; public RefundListener(RefundService refundService) { this.refundService = refundService; } @KafkaListener( id = "payment-refunds", groupId = "payment-service", topics = "payments.refund-requested", containerFactory = "manualAckKafkaListenerContainerFactory", concurrency = "1", properties = { "max.poll.records=10", "spring.json.value.default.type=com.example.payments.events.RefundRequested" }) public void onRefundRequested(ConsumerRecord<String, RefundRequested> record, Acknowledgment ack) { try { refundService.refundOnce(record.value()); ack.acknowledge(); } catch (ProviderUnavailableException ex) { // Not acknowledged. Redeliver this record and the rest of the poll after 30 seconds. ack.nack(Duration.ofSeconds(30)); } }}Why it is written this way
ConsumerRecordgives the listener key, value, partition, offset and headers in one object.Acknowledgmentis available because the chosen factory uses a manual ack mode.acknowledge()is called after the refund is recorded. WithMANUAL_IMMEDIATEthe commit happens at once, on this thread.nack(Duration)is the alternative to throwing for a failure you expect to pass: offsets acknowledged so far are committed, the remaining records of the poll are dropped and fetched again, and the listener is paused for the sleep time. It pauses the whole listener, which is acceptable here withconcurrency = "1".propertiesoverrides consumer settings for this listener only: fewer records per poll, and a different default type for the JSON deserializer.group.idandclient.idcannot be set this way. The override reaches the deserializer because it is configured by class name inapplication.yml, not passed to the factory as an instance.- Any other exception propagates to the container's error handler, which is the right place for retries with back-off and dead letter topics.
- The 30 second sleep is far below
max.poll.interval.ms. The container keeps polling while paused, so the consumer stays in the group.
package com.example.payments.kafka;import java.util.List;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import com.example.payments.events.OrderPlaced;import com.example.payments.reporting.OrderReportRepository;import com.example.payments.reporting.OrderReportRow;@Componentpublic class OrderReportingListener { private final OrderReportRepository repository; public OrderReportingListener(OrderReportRepository repository) { this.repository = repository; } @KafkaListener( id = "payment-reporting", groupId = "payment-reporting", topics = "${app.topics.orders-placed}", batch = "true", concurrency = "2", properties = "max.poll.records=500") public void onOrders(List<ConsumerRecord<String, OrderPlaced>> records) { List<OrderReportRow> rows = records.stream() .filter(record -> record.value() != null) .map(record -> new OrderReportRow( record.value().orderId(), record.value().amountMinor(), record.value().currency(), record.partition(), record.offset())) .toList(); // One multi-row upsert keyed by order id, in one database transaction. repository.upsertAll(rows); }}Why it is written this way
batch = "true"turns this listener into a batch listener on the default factory. The list is exactly what onepollreturned for this consumer, across all its partitions, in partition order within each partition.- A different
groupIdmakes this an independent reader of the same topic with its own offsets. It does not compete with the payment listener. - The container commits after the method returns, for the whole list at once. A crash before that repeats the whole list, so the write is an upsert by order id.
- The
List<ConsumerRecord<...>>parameter must be the only one, apart from an optionalAcknowledgmentorConsumer. - With
ErrorHandlingDeserializer, a record that failed to deserialise arrives in a batch with a null value and the exception in a header. The filter skips those here; a real service would count them or route them to a dead letter topic. - If the method throws, the container cannot know which record was at fault and treats the whole batch as failed. Spring error handling shows
BatchListenerFailedException, which names the failing index.
package com.example.payments.kafka;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.config.KafkaListenerEndpointRegistry;import org.springframework.kafka.listener.MessageListenerContainer;import org.springframework.stereotype.Component;/** Pauses and resumes the payment listener, for example from a circuit breaker on the card provider. */@Componentpublic class PaymentListenerControl { private static final Logger log = LoggerFactory.getLogger(PaymentListenerControl.class); private final KafkaListenerEndpointRegistry registry; public PaymentListenerControl(KafkaListenerEndpointRegistry registry) { this.registry = registry; } public void pause() { MessageListenerContainer container = container(); if (!container.isPauseRequested()) { container.pause(); log.warn("Pause requested for listener {}", PaymentListener.LISTENER_ID); } } public void resume() { MessageListenerContainer container = container(); if (container.isPauseRequested()) { container.resume(); log.info("Resume requested for listener {}", PaymentListener.LISTENER_ID); } } public boolean isPaused() { return container().isContainerPaused(); } private MessageListenerContainer container() { MessageListenerContainer container = registry.getListenerContainer(PaymentListener.LISTENER_ID); if (container == null) { throw new IllegalStateException("No listener container with id " + PaymentListener.LISTENER_ID); } return container; }}Why it is written this way
- The registry is an infrastructure bean that Spring declares; inject it like any other. The lookup uses the
idfrom the annotation. pause()is a request. It takes effect before the next poll, so records already fetched are still delivered.isPauseRequested()reports the request,isContainerPaused()reports that all consumers have actually paused.- A paused container keeps polling without receiving records, so the consumers stay in the group and no rebalance happens. Lag on their partitions grows until
resume(). - Pausing the concurrent container pauses all its consumer threads. Pause state is per application instance and is not persisted: each instance must decide for itself, and a restart begins unpaused.
- Call these methods from another thread (a health check, a circuit breaker callback, an admin endpoint). The reference recommends the container's
pauseandresumeover touching theConsumerdirectly, which is not thread-safe.
Configuration
Container defaults are from the Spring for Apache Kafka 3.3 container properties table, Boot keys from the Spring Boot 3.5 appendix, and Kafka client defaults from the 3.9 consumer configuration reference.
spring.kafka.listener.ack-modeDefault none in Spring Boot; the container default is BATCH
- What it does
- When the container commits offsets:
record,batch,time,count,count_time,manualormanual_immediate. - When to change it
recordwhen repeating a whole poll is expensive;batchfor cheap idempotent work; a manual mode only when returning from the method is not the right moment to commit.
spring.kafka.listener.concurrencyDefault none in Spring Boot; the container default is 1
- What it does
- Consumers and threads per listener in this instance. Overridden per listener by the annotation's
concurrency. - When to change it
- Instances multiplied by concurrency should not exceed the partition count. More only adds idle consumers.
spring.kafka.listener.typeDefault single
- What it does
- Whether the default factory creates record (
single) orbatchlisteners. Overridden per listener bybatch = "true"or"false". - When to change it
- Leave it at
singleand opt in per listener, so that the delivery style is visible on the method.
spring.kafka.consumer.enable-auto-commit (enable.auto.commit)Default none in Spring Boot; the Kafka client default is true, but the listener container sets it to false unless you set it
- What it does
- Whether the Kafka client commits offsets on a timer, independent of the listener.
- When to change it
- Do not set it. With true, the client can commit records the listener has not finished, and the container's ack mode no longer describes what happens.
spring.kafka.consumer.max-poll-records (max.poll.records)Default none in Spring Boot; Kafka client default 500
- What it does
- Upper bound on records returned by one poll: the amount of work between two polls, and the size of a batch listener's list.
- When to change it
- Lower it when single records are slow. Worst-case time per record multiplied by this value must stay well under
max.poll.interval.ms.
spring.kafka.consumer.max-poll-interval (max.poll.interval.ms)Default none in Spring Boot; Kafka client default 300000 (5 minutes)
- What it does
- Maximum time between two polls before the consumer is considered failed and its partitions are reassigned.
- When to change it
- Raise it only for genuinely long work, knowing that a stuck consumer is then also detected later. Prefer fewer records per poll.
spring.kafka.listener.poll-timeoutDefault none in Spring Boot; the container default is 5000 ms
- What it does
- How long one
pollcall waits for records. It also sets the resolution of pause, resume andnacksleep times. - When to change it
- Rarely changed. Lower it if
nacksleeps or pause reactions need to be more precise than five seconds.
spring.kafka.listener.async-acksDefault none in Spring Boot; the container default is false
- What it does
- Allows manual acknowledgements out of order. The container defers the commit until the gaps are filled and pauses the consumer until the previous poll is fully committed.
- When to change it
- Only for deliberate asynchronous processing with a manual ack mode. It raises the chance of duplicates after a failure and cannot be combined with
nack.
spring.kafka.listener.auto-startupDefault true
- What it does
- Whether listener containers start with the application. Overridden per listener by
autoStartup. - When to change it
- Set a listener's
autoStartupto"false"when something else (a feature flag, a completed migration) must start it through the registry.
spring.kafka.listener.immediate-stopDefault false
- What it does
- On stop, whether the container stops after the current record instead of after all records of the current poll.
- When to change it
- Enable it when a poll can take long and shutdown must be quick; unprocessed records are simply fetched again later.
session.timeout.ms / heartbeat.interval.msDefault 45000 / 3000 (Kafka client 3.9, classic protocol)
- What it does
- How long the coordinator waits without heartbeats before removing a member, and how often the client's background thread sends them.
- When to change it
- These detect a dead process. They do not detect a slow listener, which is what
max.poll.interval.msis for.
Hands-on exercise
Size concurrency for the payment listener
Treat each consumer in the simulator as one listener thread and find out what concurrency and instance count do to assignment, idle threads and lag when a thread disappears.
Steps · 0 of 7 done
Consumer group simulator: listener threads on orders.placed
Simulation · not a real clusterEach consumer here stands for one thread of a @KafkaListener container. Start with one instance at concurrency 3, then add, remove and crash consumers. The simulation is deterministic and is not a real cluster.
What you should see
With 3 consumers each owns two partitions; with 6 each owns one; consumers 7 and 8 own nothing and add no throughput. A crashed consumer's partition accumulates lag until the group notices and reassigns it, after which the new owner continues from the last committed offset. Cooperative-sticky moves only the partitions that change owner, while range revokes everything first. For 2 instances, concurrency 3 uses all six partitions; for 4 instances, concurrency 1 or 2 is enough, and anything above 6 consumers in total is idle threads and connections.
What can go wrong
Slow work on the listener thread
- Symptom
- Under load the group rebalances every few minutes, the same records are processed repeatedly, and commits fail with
CommitFailedException. - Cause
- The records of one poll take longer than
max.poll.interval.msto process. The consumer is removed from the group while still working, its partitions move, and the new owner starts from the last committed offset. - Fix
- Lower
max.poll.recordsso a poll's work fits the interval with margin, bound every remote call with a timeout, and make processing idempotent. Raisemax.poll.interval.msonly if single records are legitimately slow.
Returning before the work is done
- Symptom
- After a deployment or crash, some orders were never charged, although the consumer group shows no lag.
- Cause
- The listener submits the work to an executor (or calls an
@Asyncmethod) and returns. The container commits on return, so the offsets move past records whose work is still queued in memory. - Fix
- Process on the consumer thread. If you need asynchronous processing, use a manual ack mode with
asyncAcks, or an asynchronous return type, so the acknowledgement follows completion, and accept the ordering and duplicate consequences.
More threads than partitions
- Symptom
- Raising
concurrencyfrom 6 to 24 does not change throughput.kafka-consumer-groups.sh --describe --membersshows many members with zero partitions. - Cause
- A partition is assigned to exactly one consumer of a group. Threads beyond the partition count have nothing to read.
- Fix
- Set instances multiplied by concurrency to at most the partition count. For more parallelism add partitions (knowing the effect on keys), speed up the work or batch it.
Acknowledgment that never commits
- Symptom
- A listener calls
ack.acknowledge()on some records only, expecting the others to be redelivered. They are not, until a restart replays a large range. - Cause
- Offsets are positions. Acknowledging a later record of a partition commits past the earlier ones. Records never acknowledged at the end of a partition leave the committed offset behind, which only shows on the next rebalance or restart.
- Fix
- Acknowledge every record in order after it is handled, including those you decide to skip. To reject a record, throw or call
nack, do not just omit the acknowledgement.
Shared state in the listener bean
- Symptom
- Counters or caches kept in listener fields show impossible values, or a
HashMapfield throwsConcurrentModificationException, only when concurrency is above 1. - Cause
- One listener instance is invoked from all consumer threads of the container.
- Fix
- Keep listeners stateless, or use thread-safe structures. Per-thread state belongs in a
ThreadLocalthat is cleaned up when the consumer thread stops.
The id silently becoming the group
- Symptom
- After adding
id = "orders"to a listener, it reprocesses the topic from the configured reset position, and a new consumer group calledordersappears. - Cause
- When
groupIdis not set on the annotation, theidis used as thegroup.id, overridingspring.kafka.consumer.group-id. A new group has no committed offsets. - Fix
- Set
groupIdexplicitly on every listener that has anid, or setidIsGroup = false.
Pausing and forgetting that lag grows
- Symptom
- A listener paused during a provider outage is resumed an hour later and floods the provider with the backlog.
- Cause
- A paused consumer keeps its partitions and reads nothing. Everything produced meanwhile is waiting.
- Fix
- Alert on lag while paused, resume automatically from a health probe, and rate-limit the listener's downstream calls so that catching up is controlled.
How to troubleshoot
Repeated rebalances and duplicate processing under load
Check
The consumer log line `consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms`
- How to read it
- The listener spent too long on one poll's records. Compare your slowest record time multiplied by
max.poll.recordswith the interval. If the line is absent, the rebalances have another cause, such as restarts or missed heartbeats.
Not sure which thread owns which partition, or whether some threads are idle
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --members --verbose
- How to read it
- One row per consumer with its client id, host and assigned partitions. Members with no partitions are surplus concurrency. The client id suffix (
-0,-1, ...) matches the container's child index.
Lag grows on some partitions only
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
- How to read it
- Compare
LAGper partition and theCONSUMER-IDthat owns it. Lag on the partitions of one consumer points to a stuck or paused thread in one instance; lag on one partition with a healthy owner points to a hot key or a slow record. A-in the consumer column means nobody owns the partition.
Need to see when partitions were assigned or taken away in an instance
Check
The container's INFO log lines `partitions assigned: [...]` and `partitions revoked: [...]`, prefixed with the group id
- How to read it
- Each rebalance produces these lines per consumer thread. Frequent revoke and assign pairs with no deployment in progress mean the group is unstable; check poll interval and session timeouts next.
Records are not arriving and the application looks healthy
Check
`registry.getListenerContainer(id)`: `isRunning()`, `isPauseRequested()`, `isContainerPaused()` and `getAssignedPartitions()`
- How to read it
- Not running means the container was stopped, never started (
autoStartup), or stopped itself after a fatal error such as an authorisation failure. Paused explains growing lag with assigned partitions. An empty assignment means other members own all partitions.
The Acknowledgment parameter cannot be used
Check
The ack mode of the container factory the listener uses (`spring.kafka.listener.ack-mode` or `setAckMode` on a custom factory), and the listener's `containerFactory` attribute
- How to read it
- An
Acknowledgmentis only provided inMANUALorMANUAL_IMMEDIATEmode. A listener that names no factory useskafkaListenerContainerFactorywith the mode fromapplication.yml.
Key takeaways
- Each
@KafkaListenergets a container; each unit ofconcurrencyis one consumer and one thread that both polls and runs your method. - While your method runs, that consumer does not poll. All records of a poll must be processed within
max.poll.interval.ms, or the partitions are reassigned and the work repeats. - Spring sets
enable.auto.commit=falseand commits after the listener: per poll inBATCH(the default), per record inRECORD, or when you acknowledge in the manual modes. - An acknowledgement commits a position that covers all earlier records of the partition. Acknowledge in order, after the work.
- Every ack mode is at-least-once at best. Idempotent processing is still your job.
- Total consumers of a group beyond the partition count are idle. Size concurrency from partitions and instances.
- Pause keeps partitions and avoids a rebalance; stop leaves the group. Both are reached through
KafkaListenerEndpointRegistryby listener id.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A listener on orders.placed uses the defaults: AckMode BATCH, max.poll.records 500, max.poll.interval.ms 300000. Each record triggers a fraud check that normally takes 200 ms. During a provider slowdown each check takes 900 ms. The group starts rebalancing continuously and the same orders are checked several times.
Which change addresses the cause with the least side effect?
Scenario 2 of 3
A payment listener takes an Acknowledgment and runs with AckMode MANUAL_IMMEDIATE. For partition 2 it receives offsets 100 to 104 in one poll. Processing of 102 fails with a validation error; the code logs it and does not acknowledge 102, then processes and acknowledges 103 and 104. The team expects 102 to come back later.
What will actually happen to the record at offset 102?
Scenario 3 of 3
orders.placed has 6 partitions. The payment service runs 3 instances, each with @KafkaListener concurrency = "4". The card provider allows 12 parallel calls. Lag is rising at peak, and someone proposes raising concurrency to 8 per instance.
What does the current setup look like, and what would the change do?
References
References: Spring for Apache Kafka 3.3 reference: @KafkaListener Annotation (record and batch listeners, metadata, annotation properties) ↗ · Spring for Apache Kafka 3.3 reference: Message Listener Containers (concurrency, committing offsets, AckMode, nack) ↗ · Spring for Apache Kafka 3.3 reference: Listener Container Properties (defaults) ↗ · Spring for Apache Kafka 3.3 reference: Manually Committing Offsets (asyncAcks, out-of-order commits) ↗ · Spring for Apache Kafka 3.3 reference: Asynchronous @KafkaListener Return Types ↗ · Spring for Apache Kafka 3.3 reference: Pausing and Resuming Listener Containers ↗ · Spring for Apache Kafka 3.3 reference: @KafkaListener Lifecycle Management (KafkaListenerEndpointRegistry) ↗ · Spring for Apache Kafka 3.3 reference: Thread Safety (one listener instance on all consumer threads) ↗ · Spring for Apache Kafka 3.3 Javadoc: KafkaListener ↗ · Spring Boot 3.5: Common application properties (spring.kafka.listener.*, spring.kafka.consumer.*) ↗ · Apache Kafka 3.9: Consumer configs (max.poll.interval.ms, max.poll.records, session.timeout.ms, enable.auto.commit) ↗ · Report a problem