Skip to content

Path 9 · lesson 2 of 6

Advanced35 minJava 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 · kafka-clients 3.9 (the version Spring Boot 3.5 manages; classic group protocol by default) against Apache Kafka 4.x brokers

@KafkaListener in depth: threads, acknowledgements, batches

What runs behind @KafkaListener: the listener container and its threads, when offsets are committed in each ack mode, batch listeners, concurrency against partitions, and pausing.

What you will learn

  • Describe the container, consumer and thread that each @KafkaListener creates, and what runs on that thread
  • Choose listener method parameters for the job: payload, ConsumerRecord, headers, Acknowledgment or a batch
  • Say exactly when offsets are committed in each AckMode and what a crash at each point redelivers
  • Set concurrency from the partition count and the number of instances, and predict which consumers sit idle
  • Keep a listener inside max.poll.interval.ms, and judge when handing work to another thread is safe
  • Pause, resume, stop and start a listener at runtime through KafkaListenerEndpointRegistry

Before you start

The concept

@KafkaListener on a method looks like a callback, but it declares a small runtime. For each annotated method Spring creates a listener container. The container owns one or more Kafka consumers, runs the poll loop for each on its own thread, converts every record to your method's parameters, calls the method, and commits offsets afterwards according to its acknowledgement mode.

The single most useful fact about it: the thread that calls your method is the thread that polls. While your method runs, that consumer is not polling. Everything about speed, rebalances, pausing and commit timing follows from that.

AttributeWhat it setsNotes
topics / topicPatternThe subscription; Kafka's group management assigns partitionsPlaceholders such as ${app.topics.orders-placed} and SpEL are allowed
topicPartitionsExplicit partitions with optional start offsets, without group assignmentFor special cases such as reading a whole compacted topic on every instance
groupIdThe consumer group for this listener, overriding spring.kafka.consumer.group-idEach group receives every record once
idThe container's id, used to find it in the registry and in thread namesAlso becomes the group id when groupId is not set, unless idIsGroup = false
concurrencyNumber of consumers and threads for this listener in this instanceA string, so it can be a placeholder; overrides the factory's value
containerFactoryBean name of the container factory to useDefault is kafkaListenerContainerFactory
autoStartupWhether the container starts with the application"false" for listeners you start through the registry
batchDeliver each poll as a list instead of record by recordOverrides the factory's batch setting
propertiesKafka consumer properties for this listener only, as key=value stringsgroup.id and client.id are ignored here; use groupId and clientIdPrefix
clientIdPrefixPrefix of the consumers' client.idA suffix -n is added per consumer thread
errorHandler, filter, containerPostProcessorBean names of per-listener extensionsContainer-level error handling is the subject of the next but one lesson
The @KafkaListener attributes you will use, from the 3.3 Javadoc. Most accept property placeholders.

Why it exists

A hand-written consumer is a loop: poll, process each record, commit, repeat, on a thread you manage. It is short, and it has to get several things right at once: commit only what was processed, never touch the consumer from a second thread (KafkaConsumer is not thread-safe), keep calling poll often enough to stay in the group, handle a failing record without ending the loop, and stop cleanly on shutdown.

The listener container is that loop written once. It gives you a declarative method and takes over threading, commits, error handling hooks and lifecycle. The price is that the loop is now out of sight. People who do not know it is there make the same mistakes as before, only less visibly: slow work inside the listener, work handed to another thread while the container commits, or more threads than partitions.

How it works inside

Containers, consumers and threads

Boot's factory creates a ConcurrentMessageListenerContainer per listener. It holds concurrency child containers (KafkaMessageListenerContainer), and each child owns one KafkaConsumer and one thread. The thread name contains the listener id and the child index, for example payment-orders-0-C-1, which makes log lines attributable to a consumer.

  1. The consumer thread calls poll with the container's pollTimeout (5 seconds by default) and receives up to max.poll.records records (client default 500) from its assigned partitions.
  2. For a record listener it calls your method once per record, in order, on the same thread. For a batch listener it calls the method once with the whole list.
  3. After the listener returns, the container decides whether to commit, based on the ack mode.
  4. It polls again. Only now can it receive new records, learn about a rebalance, or act on a pause or stop request.

With the classic group protocol, heartbeats come from a separate background thread of the Kafka client, so a slow listener does not trip session.timeout.ms. What it trips is max.poll.interval.ms (default 5 minutes): if the time between two poll calls exceeds it, the consumer leaves the group, its partitions are given to others, and its next commit fails. The records it was working on are delivered again to the new owner.

One listener object serves all consumer threads. With concurrency = 3 your method runs on three threads at once, each with records from different partitions. Keep listeners stateless or thread-safe.

What the method can receive

ParameterWhat you getUse it when
The payload type, e.g. OrderPlaced (optionally @Payload)The deserialised and converted record valueThe value is all you need
ConsumerRecord<K, V>The raw record: key, value, topic, partition, offset, timestamp, headersYou log or store the record's coordinates, or need the headers
@Header(KafkaHeaders.RECEIVED_KEY) and similarOne piece of metadata: RECEIVED_KEY, RECEIVED_TOPIC, RECEIVED_PARTITION, OFFSET, RECEIVED_TIMESTAMPYou want the payload type plus one or two coordinates
ConsumerRecordMetadataEverything from the record except key and valueYou want all coordinates without the raw record
AcknowledgmentA handle to acknowledge (or nack) this record or batchThe ack mode is MANUAL or MANUAL_IMMEDIATE
Consumer<?, ?>The underlying Kafka consumerRarely: only on the listener thread, and never to seek or commit behind the container's back
List<OrderPlaced>, List<ConsumerRecord<K, V>> or ConsumerRecords<K, V>The whole poll (batch listeners)The work is cheaper in bulk, such as a JDBC batch insert
RECEIVED_KEY is absent when the record key is null, so mark that header required = false if keys can be missing.

For batch listeners the reference is strict: a List<ConsumerRecord<K, V>> or ConsumerRecords<K, V> parameter must be the only parameter apart from an optional Acknowledgment and Consumer. Annotations such as @Payload and @Header must be on the concrete method, not on an interface it implements.

Ack modes: when the commit happens

Since version 2.3 the container sets enable.auto.commit to false unless you set it yourself, because it has its own commit logic. The Kafka client's default is true, so this is a place where Spring behaves differently from a plain consumer. The container then commits according to its AckMode; commits are synchronous by default (syncCommits).

AckModeCommit happensRedelivered after a crash
BATCH (default)When all records returned by the poll have been processedEvery record of the current poll, including those already processed
RECORDAfter the listener returns from each recordAt most the record in progress
TIME / COUNT / COUNT_TIMEAfter a poll's records are processed, once ackTime has passed or ackCount records were receivedEverything since the last commit: possibly several polls
MANUALThe listener calls acknowledge(); the commit is then done as in BATCH, after the poll's recordsLike BATCH, limited to what was acknowledged
MANUAL_IMMEDIATEAt once when acknowledge() is called on the listener threadWhatever was not acknowledged
All modes are at-least-once as long as you acknowledge after the work. The modes differ in commit frequency and in how much is repeated.

RECORD narrows the duplicate window to one record for the cost of one commit request per record. BATCH is the cheapest and repeats up to a whole poll. Neither removes the need for an idempotent consumer: a crash between your database commit and the offset commit repeats the record in every mode.

The manual modes exist for cases where returning from the method is not the right moment: you acknowledge only after a downstream call confirms, or decline with nack(Duration), which commits what was acknowledged before, discards the rest of the poll and redelivers from the failed record after the given sleep. nack must be called on the listener thread, and it pauses the whole listener, not one partition. If your method takes an Acknowledgment but the container is not in a manual mode, the reference says the parameter will be null; the container factory decides the mode, not the method signature.

When the listener throws, the container's error handler takes over. By default that is a DefaultErrorHandler, which redelivers the failed record a limited number of times and then logs and skips it. Configuring it, and sending failures to a dead letter topic, is the subject of Spring error handling.

Concurrency against partitions

concurrency is the number of consumers one application instance contributes to the group for this listener. The group's total is instances multiplied by concurrency, and Kafka assigns each partition to exactly one of them. Consumers beyond the partition count receive nothing.

Five listener threads, three partitions

Simulation · not a real cluster

Think of this as one instance with concurrency = 5 on a topic with three partitions. See which consumers get a partition and which stay idle, then remove consumers one by one.

  • Six partitions, two instances, concurrency = 3: six consumers, one partition each. Adding a third instance adds three idle consumers, which is useful only as standby.
  • Six partitions, one instance, concurrency = 2: two threads with three partitions each. Records of different partitions are interleaved on a thread, but each partition is still processed in order.
  • A listener on several topics with the 3.9 client's default assignor list uses RangeAssignor, which assigns per topic. The reference's example: three topics of five partitions with concurrency = 15 leaves ten consumers idle, because each topic's five partitions go to the same first five consumers. RoundRobinAssignor or CooperativeStickyAssignor spreads them.

More threads than partitions do not add throughput. If one partition's records are slow, the options are more partitions (with the key-ordering consequences from partition sizing), faster processing, or bulk work in a batch listener.

Handing work to another thread

The tempting fix for a slow listener is to submit the work to an executor and return immediately. Know what you give up:

  • The delivery guarantee. In BATCH or RECORD mode the container commits when the method returns. The work is still in an in-memory queue. A crash loses it: at-most-once, without anyone having chosen that.
  • Ordering. Two records of the same partition can now run at the same time or finish in the opposite order.
  • Backpressure. The consumer keeps polling at full speed while the executor queue grows until memory runs out or tasks are rejected.

If you must process asynchronously, keep the commit tied to completion. Spring for Apache Kafka offers two supported ways: the container property asyncAcks with a manual ack mode, where acknowledgements may arrive in any order and the container defers commits until the gaps are filled (the consumer is paused until all offsets of the previous poll are committed); and, since 3.2, listener methods that return CompletableFuture<?> or Mono<?>, for which the container switches to manual acknowledgement with out-of-order commits and acknowledges when the future completes. The reference notes that both increase the chance of duplicate deliveries after a failure, and neither restores per-key ordering. In most services the better answer is to stay synchronous, lower max.poll.records, and scale with partitions.

Pause, resume, stop, start

Containers created for @KafkaListener are not beans. They are registered in the KafkaListenerEndpointRegistry, where getListenerContainer(id) returns one by the annotation's id.

pause() and resume()

  • The consumer stays in the group and keeps its partitions
  • The container keeps calling poll, so no rebalance happens, but no records are returned
  • Takes effect before the next poll: records of the current poll are still delivered, unless pauseImmediate is set
  • Lag grows on this consumer's partitions; nobody else picks them up
  • For short dependency outages and backpressure

stop() and start()

  • The consumers close and leave the group
  • A rebalance gives their partitions to the remaining members, if there are any
  • By default the records of the current poll are processed first; stopImmediate stops after the current record
  • start() rejoins and triggers another rebalance
  • For maintenance, feature switches and listeners with autoStartup = "false"

Architecture

Drawing diagram…

A flowchart. A KafkaListener method with id payment-orders and concurrency 3 is registered in the KafkaListenerEndpointRegistry, and the container factory builds a ConcurrentMessageListenerContainer for it. That container has three child containers, each with one thread and one consumer, named payment-orders-0-C-1 to payment-orders-2-C-1. The topic orders.placed has six partitions, two assigned to each child. All three threads call the same PaymentListener bean, after which the container commits offsets for the group payment-service according to the ack mode.

flowchart TB
  Ann["KafkaListener method, id payment-orders, concurrency 3"] --> Reg["KafkaListenerEndpointRegistry"]
  Fac["kafkaListenerContainerFactory"] --> CMLC
  Reg --> CMLC["ConcurrentMessageListenerContainer payment-orders"]
  subgraph Children["Three child containers, each one thread and one consumer"]
    C0["payment-orders-0-C-1"]
    C1["payment-orders-1-C-1"]
    C2["payment-orders-2-C-1"]
  end
  CMLC --> C0
  CMLC --> C1
  CMLC --> C2
  subgraph Topic["orders.placed, 6 partitions"]
    P01["Partitions 0 and 1"]
    P23["Partitions 2 and 3"]
    P45["Partitions 4 and 5"]
  end
  P01 --> C0
  P23 --> C1
  P45 --> C2
  C0 --> Bean["One PaymentListener bean, called from all three threads"]
  C1 --> Bean
  C2 --> Bean
  Bean --> Ack["Container commits offsets per AckMode"]
  Ack --> Off["Group payment-service offsets"]
One annotated method, one concurrent container, three consumer threads sharing six partitions, one listener bean.
  • The annotation is only a declaration. The registry holds the container, and the id is how you reach it at runtime.
  • Each child container is an independent group member with its own consumer, poll loop and assigned partitions. A slow record on one thread delays only that thread's partitions.
  • There is one listener bean. Instance fields are shared by all threads.
  • Commits are made by the same thread that polled and processed, after the listener returns. Nothing commits in the background.

Message flow

Drawing diagram…

A sequence diagram. The consumer thread polls and receives offsets 40 and 41 of orders.placed partition 0. It calls the listener for offset 40, the listener inserts a payment and returns, and the thread commits offset 41. It then calls the listener for offset 41. The database is slow and the insert takes six minutes, during which the thread does not poll. The max poll interval passes and partition 0 is reassigned. When the listener returns, the commit of offset 42 fails because the member no longer owns the partition, and offset 41 is delivered again to the new owner.

sequenceDiagram
  participant B as Broker and group coordinator
  participant T as Consumer thread payment-orders-0-C-1
  participant L as PaymentListener
  participant D as Payments database
  T->>B: poll
  B-->>T: orders.placed-0 offsets 40 and 41
  T->>L: onOrderPlaced offset 40
  L->>D: insert payment for order o-40
  D-->>L: committed
  L-->>T: return
  T->>B: commit offset 41, AckMode RECORD
  T->>L: onOrderPlaced offset 41
  L->>D: insert payment for order o-41
  Note over L,D: slow dependency, the thread is not polling
  D-->>L: committed after 6 minutes
  L-->>T: return
  Note over B,T: max.poll.interval.ms passed, partition 0 reassigned
  T->>B: commit offset 42
  B-->>T: commit fails, member no longer owns the partition
  B-->>T: offset 41 is delivered again to the new owner
AckMode RECORD on one thread: the normal path, and a record that takes longer than max.poll.interval.ms.
  1. In RECORD mode the commit after offset 40 is for offset 41: the position of the next record to read.
  2. While the listener waits for the database, nothing else happens on this thread. No poll, no commit, no reaction to a pause or stop request.
  3. After max.poll.interval.ms the group gives the partition to another member. The payment for o-41 was already written, the commit fails, and the new owner processes o-41 again.
  4. The second processing must be harmless. That is the job of an idempotent consumer, not of any listener setting.

In a real system

The payment service of the example project reads orders.placed (6 partitions, key = order id) in the group payment-service. It runs two instances with concurrency = 3, so each of the six consumers owns one partition. Charging a card is a remote call that usually takes 150 ms and occasionally several seconds.

DecisionValueReason
Ack modeRECORDA redelivered poll of 50 charge requests is expensive to deduplicate; one record is not
max.poll.records5050 records at a worst case of 3 s each is 150 s, inside the 300 s interval
max.poll.interval.ms300000 (default)Left alone: raising it also delays the detection of a stuck consumer
Concurrency3 per instance, 2 instancesEquals the six partitions; a third instance would be standby only
Provider outagePause the containerStops consumption without leaving the group; lag grows visibly and drains after resume
DuplicatesUnique constraint on order id in the payments tableRedelivery after a crash or rebalance cannot charge twice

A second listener in the same service writes every order event to a reporting table. Nobody waits for it and the cost is the database round trip, so it is a batch listener in its own group payment-reporting: one multi-row insert per poll, and one commit per poll.

Partitions & consumer groups

Simulation, not a real cluster

The animated Build lesson shows partitions being shared within a group and what an extra consumer does.

Step 1 · The pattern

A log and a reader

The producer appends messages to a partition: an ordered log. A consumer reads them one after another, in the order they were written.

  • message
  • consumed
Open the full animated lesson

Java and Spring Boot

Three listeners of the payment service and the code around them: a record listener with metadata, a manual-acknowledgement variant, a batch listener, the extra container factory, and runtime control through the registry. Written against the Spring for Apache Kafka 3.3 reference; not compiled in this environment.

src/main/resources/application.yml (consumer and listener)YAML
spring:  kafka:    consumer:      group-id: payment-service      auto-offset-reset: earliest      max-poll-records: 50      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer      properties:        "[spring.deserializer.value.delegate.class]": org.springframework.kafka.support.serializer.JsonDeserializer        "[spring.json.value.default.type]": com.example.payments.events.OrderPlaced        "[spring.json.use.type.headers]": false        "[partition.assignment.strategy]": org.apache.kafka.clients.consumer.CooperativeStickyAssignor    listener:      ack-mode: record      concurrency: 3      missing-topics-fatal: trueapp:  topics:    orders-placed: orders.placed

Why it is written this way

  • spring.kafka.listener.* configures Boot's default container factory. ack-mode: record and concurrency: 3 apply to every listener that does not override them.
  • enable.auto.commit is not mentioned. The container sets it to false and commits itself. Setting it to true here would add timer-based commits that ignore whether your listener has finished.
  • max-poll-records: 50 is chosen from the processing time, as in the table above. It limits how much work must fit into max.poll.interval.ms and how many records a BATCH mode crash repeats.
  • The deserializer lines are explained in Spring serialisation.

Configuration

Container defaults are from the Spring for Apache Kafka 3.3 container properties table, Boot keys from the Spring Boot 3.5 appendix, and Kafka client defaults from the 3.9 consumer configuration reference.

spring.kafka.listener.ack-modeDefault none in Spring Boot; the container default is BATCH
What it does
When the container commits offsets: record, batch, time, count, count_time, manual or manual_immediate.
When to change it
record when repeating a whole poll is expensive; batch for cheap idempotent work; a manual mode only when returning from the method is not the right moment to commit.
spring.kafka.listener.concurrencyDefault none in Spring Boot; the container default is 1
What it does
Consumers and threads per listener in this instance. Overridden per listener by the annotation's concurrency.
When to change it
Instances multiplied by concurrency should not exceed the partition count. More only adds idle consumers.
spring.kafka.listener.typeDefault single
What it does
Whether the default factory creates record (single) or batch listeners. Overridden per listener by batch = "true" or "false".
When to change it
Leave it at single and opt in per listener, so that the delivery style is visible on the method.
spring.kafka.consumer.enable-auto-commit (enable.auto.commit)Default none in Spring Boot; the Kafka client default is true, but the listener container sets it to false unless you set it
What it does
Whether the Kafka client commits offsets on a timer, independent of the listener.
When to change it
Do not set it. With true, the client can commit records the listener has not finished, and the container's ack mode no longer describes what happens.
spring.kafka.consumer.max-poll-records (max.poll.records)Default none in Spring Boot; Kafka client default 500
What it does
Upper bound on records returned by one poll: the amount of work between two polls, and the size of a batch listener's list.
When to change it
Lower it when single records are slow. Worst-case time per record multiplied by this value must stay well under max.poll.interval.ms.
spring.kafka.consumer.max-poll-interval (max.poll.interval.ms)Default none in Spring Boot; Kafka client default 300000 (5 minutes)
What it does
Maximum time between two polls before the consumer is considered failed and its partitions are reassigned.
When to change it
Raise it only for genuinely long work, knowing that a stuck consumer is then also detected later. Prefer fewer records per poll.
spring.kafka.listener.poll-timeoutDefault none in Spring Boot; the container default is 5000 ms
What it does
How long one poll call waits for records. It also sets the resolution of pause, resume and nack sleep times.
When to change it
Rarely changed. Lower it if nack sleeps or pause reactions need to be more precise than five seconds.
spring.kafka.listener.async-acksDefault none in Spring Boot; the container default is false
What it does
Allows manual acknowledgements out of order. The container defers the commit until the gaps are filled and pauses the consumer until the previous poll is fully committed.
When to change it
Only for deliberate asynchronous processing with a manual ack mode. It raises the chance of duplicates after a failure and cannot be combined with nack.
spring.kafka.listener.auto-startupDefault true
What it does
Whether listener containers start with the application. Overridden per listener by autoStartup.
When to change it
Set a listener's autoStartup to "false" when something else (a feature flag, a completed migration) must start it through the registry.
spring.kafka.listener.immediate-stopDefault false
What it does
On stop, whether the container stops after the current record instead of after all records of the current poll.
When to change it
Enable it when a poll can take long and shutdown must be quick; unprocessed records are simply fetched again later.
session.timeout.ms / heartbeat.interval.msDefault 45000 / 3000 (Kafka client 3.9, classic protocol)
What it does
How long the coordinator waits without heartbeats before removing a member, and how often the client's background thread sends them.
When to change it
These detect a dead process. They do not detect a slow listener, which is what max.poll.interval.ms is for.

Hands-on exercise

Size concurrency for the payment listener

Treat each consumer in the simulator as one listener thread and find out what concurrency and instance count do to assignment, idle threads and lag when a thread disappears.

Steps · 0 of 7 done

Consumer group simulator: listener threads on orders.placed

Simulation · not a real cluster

Each consumer here stands for one thread of a @KafkaListener container. Start with one instance at concurrency 3, then add, remove and crash consumers. The simulation is deterministic and is not a real cluster.

Take it further in the lab: Consumer group assignment

What can go wrong

Slow work on the listener thread
Symptom
Under load the group rebalances every few minutes, the same records are processed repeatedly, and commits fail with CommitFailedException.
Cause
The records of one poll take longer than max.poll.interval.ms to process. The consumer is removed from the group while still working, its partitions move, and the new owner starts from the last committed offset.
Fix
Lower max.poll.records so a poll's work fits the interval with margin, bound every remote call with a timeout, and make processing idempotent. Raise max.poll.interval.ms only if single records are legitimately slow.
Returning before the work is done
Symptom
After a deployment or crash, some orders were never charged, although the consumer group shows no lag.
Cause
The listener submits the work to an executor (or calls an @Async method) and returns. The container commits on return, so the offsets move past records whose work is still queued in memory.
Fix
Process on the consumer thread. If you need asynchronous processing, use a manual ack mode with asyncAcks, or an asynchronous return type, so the acknowledgement follows completion, and accept the ordering and duplicate consequences.
More threads than partitions
Symptom
Raising concurrency from 6 to 24 does not change throughput. kafka-consumer-groups.sh --describe --members shows many members with zero partitions.
Cause
A partition is assigned to exactly one consumer of a group. Threads beyond the partition count have nothing to read.
Fix
Set instances multiplied by concurrency to at most the partition count. For more parallelism add partitions (knowing the effect on keys), speed up the work or batch it.
Acknowledgment that never commits
Symptom
A listener calls ack.acknowledge() on some records only, expecting the others to be redelivered. They are not, until a restart replays a large range.
Cause
Offsets are positions. Acknowledging a later record of a partition commits past the earlier ones. Records never acknowledged at the end of a partition leave the committed offset behind, which only shows on the next rebalance or restart.
Fix
Acknowledge every record in order after it is handled, including those you decide to skip. To reject a record, throw or call nack, do not just omit the acknowledgement.
Shared state in the listener bean
Symptom
Counters or caches kept in listener fields show impossible values, or a HashMap field throws ConcurrentModificationException, only when concurrency is above 1.
Cause
One listener instance is invoked from all consumer threads of the container.
Fix
Keep listeners stateless, or use thread-safe structures. Per-thread state belongs in a ThreadLocal that is cleaned up when the consumer thread stops.
The id silently becoming the group
Symptom
After adding id = "orders" to a listener, it reprocesses the topic from the configured reset position, and a new consumer group called orders appears.
Cause
When groupId is not set on the annotation, the id is used as the group.id, overriding spring.kafka.consumer.group-id. A new group has no committed offsets.
Fix
Set groupId explicitly on every listener that has an id, or set idIsGroup = false.
Pausing and forgetting that lag grows
Symptom
A listener paused during a provider outage is resumed an hour later and floods the provider with the backlog.
Cause
A paused consumer keeps its partitions and reads nothing. Everything produced meanwhile is waiting.
Fix
Alert on lag while paused, resume automatically from a health probe, and rate-limit the listener's downstream calls so that catching up is controlled.

How to troubleshoot

Repeated rebalances and duplicate processing under load

Check

The consumer log line `consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms`
How to read it
The listener spent too long on one poll's records. Compare your slowest record time multiplied by max.poll.records with the interval. If the line is absent, the rebalances have another cause, such as restarts or missed heartbeats.
Not sure which thread owns which partition, or whether some threads are idle

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --members --verbose
How to read it
One row per consumer with its client id, host and assigned partitions. Members with no partitions are surplus concurrency. The client id suffix (-0, -1, ...) matches the container's child index.
Lag grows on some partitions only

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
Compare LAG per partition and the CONSUMER-ID that owns it. Lag on the partitions of one consumer points to a stuck or paused thread in one instance; lag on one partition with a healthy owner points to a hot key or a slow record. A - in the consumer column means nobody owns the partition.
Need to see when partitions were assigned or taken away in an instance

Check

The container's INFO log lines `partitions assigned: [...]` and `partitions revoked: [...]`, prefixed with the group id
How to read it
Each rebalance produces these lines per consumer thread. Frequent revoke and assign pairs with no deployment in progress mean the group is unstable; check poll interval and session timeouts next.
Records are not arriving and the application looks healthy

Check

`registry.getListenerContainer(id)`: `isRunning()`, `isPauseRequested()`, `isContainerPaused()` and `getAssignedPartitions()`
How to read it
Not running means the container was stopped, never started (autoStartup), or stopped itself after a fatal error such as an authorisation failure. Paused explains growing lag with assigned partitions. An empty assignment means other members own all partitions.
The Acknowledgment parameter cannot be used

Check

The ack mode of the container factory the listener uses (`spring.kafka.listener.ack-mode` or `setAckMode` on a custom factory), and the listener's `containerFactory` attribute
How to read it
An Acknowledgment is only provided in MANUAL or MANUAL_IMMEDIATE mode. A listener that names no factory uses kafkaListenerContainerFactory with the mode from application.yml.

Key takeaways

  • Each @KafkaListener gets a container; each unit of concurrency is one consumer and one thread that both polls and runs your method.
  • While your method runs, that consumer does not poll. All records of a poll must be processed within max.poll.interval.ms, or the partitions are reassigned and the work repeats.
  • Spring sets enable.auto.commit=false and commits after the listener: per poll in BATCH (the default), per record in RECORD, or when you acknowledge in the manual modes.
  • An acknowledgement commits a position that covers all earlier records of the partition. Acknowledge in order, after the work.
  • Every ack mode is at-least-once at best. Idempotent processing is still your job.
  • Total consumers of a group beyond the partition count are idle. Size concurrency from partitions and instances.
  • Pause keeps partitions and avoids a rebalance; stop leaves the group. Both are reached through KafkaListenerEndpointRegistry by listener id.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A listener on orders.placed uses the defaults: AckMode BATCH, max.poll.records 500, max.poll.interval.ms 300000. Each record triggers a fraud check that normally takes 200 ms. During a provider slowdown each check takes 900 ms. The group starts rebalancing continuously and the same orders are checked several times.

Which change addresses the cause with the least side effect?

Scenario 2 of 3

A payment listener takes an Acknowledgment and runs with AckMode MANUAL_IMMEDIATE. For partition 2 it receives offsets 100 to 104 in one poll. Processing of 102 fails with a validation error; the code logs it and does not acknowledge 102, then processes and acknowledges 103 and 104. The team expects 102 to come back later.

What will actually happen to the record at offset 102?

Scenario 3 of 3

orders.placed has 6 partitions. The payment service runs 3 instances, each with @KafkaListener concurrency = "4". The card provider allows 12 parallel calls. Lag is rising at peak, and someone proposes raising concurrency to 8 per instance.

What does the current setup look like, and what would the change do?

References

References: Spring for Apache Kafka 3.3 reference: @KafkaListener Annotation (record and batch listeners, metadata, annotation properties) ↗ · Spring for Apache Kafka 3.3 reference: Message Listener Containers (concurrency, committing offsets, AckMode, nack) ↗ · Spring for Apache Kafka 3.3 reference: Listener Container Properties (defaults) ↗ · Spring for Apache Kafka 3.3 reference: Manually Committing Offsets (asyncAcks, out-of-order commits) ↗ · Spring for Apache Kafka 3.3 reference: Asynchronous @KafkaListener Return Types ↗ · Spring for Apache Kafka 3.3 reference: Pausing and Resuming Listener Containers ↗ · Spring for Apache Kafka 3.3 reference: @KafkaListener Lifecycle Management (KafkaListenerEndpointRegistry) ↗ · Spring for Apache Kafka 3.3 reference: Thread Safety (one listener instance on all consumer threads) ↗ · Spring for Apache Kafka 3.3 Javadoc: KafkaListener ↗ · Spring Boot 3.5: Common application properties (spring.kafka.listener.*, spring.kafka.consumer.*) ↗ · Apache Kafka 3.9: Consumer configs (max.poll.interval.ms, max.poll.records, session.timeout.ms, enable.auto.commit) ↗ · Report a problem