Path 4 · lesson 4 of 5
Consumer rebalancing
What starts a rebalance, what each protocol and assignor revokes, what the three timeouts detect, and what happens to work in progress when a partition moves.
What you will learn
- List every event that starts a rebalance and say how quickly the coordinator notices each one
- Explain the JoinGroup and SyncGroup rounds of the classic protocol and why one slow member can hold up the whole group
- Say exactly which partitions an eager assignor and the cooperative sticky assignor revoke, and switch between them without breaking the group
- Choose heartbeat interval, session timeout and max poll interval from what each one detects
- Decide when static membership helps and what it costs when a member is really dead
- Commit safely when partitions are revoked, and handle CommitFailedException and RebalanceInProgressException correctly
Before you start
The concept
A consumer group promises that each partition of the subscribed topics is owned by at most one member at a time. A rebalance is the procedure that changes who owns what while keeping that promise. One broker, the group coordinator, runs it for the group. It is the broker that leads the partition of the internal __consumer_offsets topic that the group id hashes to.
A rebalance is not free. A partition that is changing owner is read by nobody until the hand-over finishes, and anything its old owner processed but had not committed will be processed again by the new owner. How much of the group stops, and for how long, depends on three things this lesson takes apart: what triggered it, which protocol and assignor the group uses, and how the timeouts are set.
| Trigger | How the coordinator learns of it | How quickly |
|---|---|---|
| A member joins (scale up, deploy, restart) | The new member sends a join request from inside poll() | Immediately |
| A member leaves cleanly | close() sends a leave request. Static members do not send one by default | Immediately for dynamic members |
| A member crashes or loses its network | No heartbeat arrives for session.timeout.ms | After the session timeout, 45 s by default |
A member stops calling poll() | The client itself notices max.poll.interval.ms has passed and leaves the group | After the max poll interval, 5 min by default |
| The subscription changes | A member rejoins with a different topic list, or a pattern subscription matches a new topic | At the next rejoin or metadata refresh |
| The partition count of a subscribed topic changes | Classic clients see it at a metadata refresh (metadata.max.age.ms, 5 min) and rejoin | Up to the metadata age |
| The coordinator broker changes | Members get an error, look up the new coordinator and continue with it | Depends on the failover, see below |
One group, six partitions
Simulation · not a real clusterAdd a consumer, then crash one. Switch the assignor between range and cooperative-sticky and compare how many partitions stop each time.
Why it exists
Without group management you would call assign() and hard-code which instance reads which partition. That works until an instance dies: its partitions are read by nobody until a person intervenes, and adding an instance means redeploying all of them with a new mapping. Rebalancing exists so that membership can change at any time, including by failure, and ownership follows automatically.
The first design, now called eager, was the simplest one that is obviously safe: before the group computes a new assignment, every member gives up everything it owns. No partition can have two owners because for a moment no partition has any owner. The price is a stop-the-world pause for the whole group on every membership change, however small. A rolling deploy of ten instances produces at least twenty such pauses.
Each later addition removes one cost of that design. A background heartbeat thread with a separate max.poll.interval.ms (KIP-62) stopped slow processing from being mistaken for a crash. Static membership (KIP-345) stopped a quick restart from counting as a leave and a join. Cooperative rebalancing (KIP-429) stopped members from revoking partitions that were never going to move. The consumer group protocol (KIP-848) moved the assignment into the broker so that members no longer have to stop and wait for each other at all.
Rebalance storms
Simulation, not a real clusterAn animated walk-through of how repeated rebalances feed on themselves, and how the timeouts stop the loop.
Step 1 · The pattern
Fetch, process, commit, repeat
The consumer that owns a partition polls for a batch, works through it, commits, and polls again. Each poll restarts a countdown. A second group member stands by.
- record
- processed, not committed
- committed
How it works inside
The classic protocol: two rounds through the coordinator
- Notification. The coordinator marks the group as rebalancing. Existing members find out from their next heartbeat response, so within
heartbeat.interval.ms. - JoinGroup. Every member sends a JoinGroup request with its subscription, the assignors it supports and, for cooperative assignors, the partitions it currently owns. The coordinator waits until all known members have rejoined or the rebalance timeout expires. In the Java client the rebalance timeout is
max.poll.interval.ms. Members that have not rejoined by then are removed. - Leader election and assignment. The coordinator picks one assignor that every member supports, names one member the group leader and sends it the member list. The leader runs the assignor in the client. The broker never computes a classic assignment.
- SyncGroup. Every member sends SyncGroup. The leader's request carries the full assignment, and the coordinator answers each member with its own share. The group generation goes up by one and the group is stable again.
Eager and cooperative: what gets revoked
Eager: range, round robin, sticky
- Before sending JoinGroup, each member revokes all its partitions.
onPartitionsRevokedreceives the full assignment, even on a member whose assignment will not change. - Nothing is consumed anywhere in the group until SyncGroup completes.
- One rebalance per change.
StickyAssignorkeeps the same owners where it can, which saves state rebuilding, but it still revokes everything first.
Cooperative: cooperative sticky
- Members keep their partitions and report them in JoinGroup.
poll()keeps returning records for them during the rebalance. - Round one: the leader computes the target. A partition that must move is removed from its current owner's assignment but not yet given to the new owner.
- Only the members that lost something call
onPartitionsRevoked, with only those partitions, and immediately rejoin. - Round two: the now free partitions are assigned. Two short rebalances instead of one long pause.
The coordinator can only choose an assignor that every member lists in partition.assignment.strategy. That rule decides how you switch. The default list is RangeAssignor, CooperativeStickyAssignor: the group uses range, and one rolling restart that removes range from the list moves it to cooperative. If you have set an eager assignor explicitly, it takes two rolling restarts: first add CooperativeStickyAssignor to the list while keeping the old one, then remove the old one. A member that lists only cooperative sticky cannot join a group whose other members list only an eager assignor, because no common assignor exists.
Three timers, three different failures
| Setting (default) | Who enforces it | What it detects |
|---|---|---|
heartbeat.interval.ms (3 s) | Client, background thread | Nothing by itself. It is how often the member says it is alive, and how quickly it hears that a rebalance has started. |
session.timeout.ms (45 s) | Coordinator | A process that is gone: crash, kill, network partition, a pause long enough to stop the heartbeat thread. |
max.poll.interval.ms (5 min) | Client | A process that is alive but stuck: the heartbeat thread still runs while the application thread never returns to poll(). |
When the poll interval is exceeded, a dynamic member leaves the group at once and its partitions are reassigned. A static member behaves differently: it stops heartbeating, and its partitions are reassigned only after the session timeout as well.
Static membership
Setting group.instance.id to a value that is unique and stable per instance makes the member static. It does not send a leave request on close(), and when it comes back with the same id inside the session timeout the coordinator hands it the same partitions without a rebalance. During the gap those partitions are simply not consumed. That is the trade: a restart costs a short pause on one member's partitions instead of two group rebalances, but a member that is really dead is noticed only after the session timeout, and to cover a restart you usually have to raise that timeout. Removing an instance for good has the same delay unless it closes with the leave option or an operator removes it through the Admin API. Two live processes with the same id are not tolerated: one of them is fenced with FencedInstanceIdException.
Work in progress and commits
- Revoked means an orderly hand-over.
onPartitionsRevokedruns insidepoll()before the partition is given away, and the member is still allowed to commit. Commit the offsets of what you have finished there, and only for the partitions passed in. - Lost means the member has already been removed, for example after a session timeout.
onPartitionsLostis called instead. Another member may own the partitions already, so a commit would be rejected. Drop local state and do not commit. CommitFailedExceptionis the fatal form: the member is no longer in the group or no longer owns the partition, most often because processing ran pastmax.poll.interval.ms. The uncommitted records will be delivered again to the new owner. Do not retry the commit.RebalanceInProgressExceptionis not fatal. It appears with cooperative rebalancing whencommitSyncis called while a rebalance is under way. Callpoll()to let it finish, then commit again for the partitions you still own.
The consumer group protocol (KIP-848)
Kafka 4.0 made a second protocol generally available, selected with group.protocol=consumer. The default in 4.x is still classic, and the project's published plan is to change the client default in 5.0. Under it the broker computes the assignment with a server-side assignor (uniform by default, or range), and a single heartbeat request carries everything: the member's subscription and owned partitions go up, its target assignment comes back. There is no JoinGroup, no SyncGroup, no group leader and no point where the whole group must be in step. Each member is moved to the new assignment on its own, and a partition is offered to its new owner only after the old owner has confirmed giving it up. A slow member delays only its own partitions.
heartbeat.interval.ms,session.timeout.msandpartition.assignment.strategyare not usable under it. The broker settingsgroup.consumer.heartbeat.interval.ms(5 s) andgroup.consumer.session.timeout.ms(45 s) take their place, andgroup.remote.assignorpicks the server-side assignor.max.poll.interval.msandgroup.instance.idstill apply. The listener callbacks behave like the cooperative case: only the partitions that move are passed in.- Custom client-side assignors are not supported, and the 4.4 documentation lists rack-aware assignment as not fully supported yet.
- The Java client supports it for production from 4.0. Spring Boot 3.x manages a 3.9 client, so stay on the classic protocol with the cooperative sticky assignor there. Spring for Apache Kafka documents support for the new protocol from its 4.0 line.
Architecture
Drawing diagram…
State diagram of a classic group: Empty, PreparingRebalance while members rejoin, CompletingRebalance while the leader's assignment is distributed, then Stable. Any membership or subscription change sends Stable back to PreparingRebalance.
stateDiagram-v2 [*] --> Empty Empty --> PreparingRebalance: first member joins PreparingRebalance --> CompletingRebalance: all members rejoined or rebalance timeout CompletingRebalance --> Stable: leader sent the assignment in SyncGroup Stable --> PreparingRebalance: join, leave, session timeout, poll timeout, subscription or partition change CompletingRebalance --> PreparingRebalance: membership changed again Stable --> Empty: last member left PreparingRebalance --> Empty: nobody rejoined Empty --> Dead: offsets expired or group deleted Dead --> [*]
- PreparingRebalance is the JoinGroup round. The group stays here until every known member has rejoined or the rebalance timeout, which the Java client sets to
max.poll.interval.ms, runs out. - CompletingRebalance is the SyncGroup round: the coordinator is waiting for the leader's assignment. It is normally very short.
- A group that never settles in Stable is in a rebalance loop. Each pass through the cycle raises the generation, and commits from an older generation are rejected.
- With an eager assignor nothing is consumed between leaving Stable and returning to it. With cooperative sticky, members keep reading the partitions they retain throughout.
Message flow
Drawing diagram…
Sequence diagram. Consumer 3 joins, consumers 1 and 2 rejoin while keeping their partitions, the leader's first assignment takes P2 and P5 away without giving them to anyone, the two owners revoke them and rejoin, and the second round gives P2 and P5 to consumer 3.
sequenceDiagram participant C1 as Consumer 1 participant C2 as Consumer 2 participant C3 as Consumer 3 new participant GC as Group coordinator C3->>GC: JoinGroup with empty owned list GC-->>C1: Heartbeat response says rebalance in progress GC-->>C2: Heartbeat response says rebalance in progress C1->>GC: JoinGroup owning P0 P1 P2 C2->>GC: JoinGroup owning P3 P4 P5 GC-->>C1: JoinGroup response, you are leader, member list Note over C1: Target is two partitions each, so P2 and P5 must move C1->>GC: SyncGroup with assignment that leaves P2 and P5 unowned C2->>GC: SyncGroup C3->>GC: SyncGroup GC-->>C1: Keep P0 P1 GC-->>C2: Keep P3 P4 GC-->>C3: Nothing yet Note over C1,C2: Each commits and revokes one partition, then rejoins C1->>GC: JoinGroup owning P0 P1 C2->>GC: JoinGroup owning P3 P4 C3->>GC: JoinGroup with empty owned list GC-->>C3: After second SyncGroup, you own P2 and P5
- Consumers 1 and 2 learn about the rebalance from a heartbeat response, then rejoin from their next
poll(). They keep reading P0, P1, P3 and P4 the whole time. - The first assignment never gives P2 or P5 to consumer 3. A partition gets a new owner only after its old owner has revoked it, which is what keeps two members from reading it at once.
- The revoke callback on consumers 1 and 2 receives one partition each. That is the place to commit the offsets for it.
- With the range assignor the same event is one round, but all six partitions are revoked before the first JoinGroup and nothing is read until the round completes.
In a real system
A typical payment service consumes orders.events: 12 partitions, keyed by order id so that the events of one order stay in order. It runs as 4 pods in a Kubernetes StatefulSet, each with a listener concurrency of 3, so the group payment-service has 12 members and each owns one partition. Handling one event means a call to a payment provider that normally takes 200 ms and occasionally 10 s.
With the range assignor and default settings, a rolling deploy restarts 4 pods one after another. Each stop is 3 leave requests and each start is 3 joins, and every one of those rebalances revokes all 12 partitions. Any pod that is in the middle of a slow provider call holds the JoinGroup round open until it returns to poll(). The deploy shows up on the lag graph as a row of spikes, and the payment provider sees a burst of repeated requests after each one.
- Cooperative sticky reduces each of those rebalances to the partitions that actually move.
- Static membership, with the pod name as
group.instance.id(Spring appends-1,-2,-3for the three listener threads), removes the rebalances altogether when a pod is back within the session timeout. The cost is that the pod's 3 partitions wait for it while it restarts, and wait the full session timeout if it never comes back. max.poll.recordsis lowered so that a worst-case batch fits comfortably insidemax.poll.interval.ms: 20 records at 10 s each is 200 s, under the 300 s default. With the default of 500 records the worst case would be far past it.- The handler stores the order id of each completed payment, so a redelivered event after a rebalance is recognised and skipped.
Java and Spring Boot
The Spring Boot configuration for the payment service above, a rebalance listener that makes ownership changes visible, and the same commit-on-revoke logic written against the plain consumer so you can see what the container does for you.
spring: kafka: bootstrap-servers: localhost:9092 consumer: group-id: payment-service enable-auto-commit: false auto-offset-reset: earliest max-poll-records: 20 key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer properties: spring.json.trusted.packages: com.example.payments spring.json.value.default.type: com.example.payments.OrderEvent # Classic protocol. Listing only this assignor makes the group cooperative. partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor heartbeat.interval.ms: 3000 session.timeout.ms: 60000 max.poll.interval.ms: 300000 # Static membership: stable per pod, for example payment-service-2 group.instance.id: ${HOSTNAME} listener: concurrency: 3 ack-mode: recordWhy it is written this way
partition.assignment.strategylists only the cooperative sticky assignor. Roll this out to a group that already supports it (the default list does); from an explicitly eager assignor go through the intermediate step of listing both.max-poll-records: 20is chosen from the worst-case handling time, so that one batch always finishes well insidemax.poll.interval.ms.session.timeout.msis raised to 60 s so that a pod restart fits inside it. That is also how long a dead pod's partitions wait, which is the cost of static membership.group.instance.iduses the pod host name, which is stable in a StatefulSet. Withconcurrency: 3Spring for Apache Kafka adds a-nsuffix per consumer so the ids stay unique.ack-mode: recordcommits after each record, which keeps the redelivery window after a lost partition to a single record.
package com.example.payments;import java.util.Collection;import org.apache.kafka.clients.consumer.Consumer;import org.apache.kafka.common.TopicPartition;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.listener.ConsumerAwareRebalanceListener;import org.springframework.stereotype.Component;/** * Spring Boot applies a single ConsumerAwareRebalanceListener bean to the * auto-configured listener container factory. */@Componentpublic class RebalanceLogger implements ConsumerAwareRebalanceListener { private static final Logger log = LoggerFactory.getLogger(RebalanceLogger.class); @Override public void onPartitionsRevokedBeforeCommit(Consumer<?, ?> consumer, Collection<TopicPartition> partitions) { // Orderly hand-over. The container commits pending offsets right after this returns. log.info("Revoking {} (still owned, commit allowed)", partitions); } @Override public void onPartitionsRevokedAfterCommit(Consumer<?, ?> consumer, Collection<TopicPartition> partitions) { for (TopicPartition partition : partitions) { log.info("Revoked {} at position {}", partition, consumer.position(partition)); } } @Override public void onPartitionsAssigned(Consumer<?, ?> consumer, Collection<TopicPartition> partitions) { // Called after every rebalance, with an empty collection if nothing new arrived. log.info("Assigned {}", partitions); } @Override public void onPartitionsLost(Consumer<?, ?> consumer, Collection<TopicPartition> partitions) { // The member was removed before it could hand over. Another member may own // these partitions already, so there is nothing left to commit. log.warn("Lost {} without a hand-over: uncommitted records will be redelivered", partitions); }}Why it is written this way
- The container already commits pending offsets when partitions are revoked. The two revoke callbacks run before and after that commit; use the first to finish in-flight work, the second to read the final position.
onPartitionsLostis separate on purpose. Do not call your revoke logic from it: a commit there would fail, and the container already runs its own clean-up for lost partitions.- Log lines like these are the fastest way to tell a rolling restart (revoked, then assigned) from a timeout (lost).
- Under the consumer group protocol
onPartitionsAssignedcan be called several times with small sets, because partitions arrive as their previous owners release them.
package com.example.payments.plain;import java.time.Duration;import java.util.Collection;import java.util.HashMap;import java.util.List;import java.util.Map;import org.apache.kafka.clients.consumer.CommitFailedException;import org.apache.kafka.clients.consumer.ConsumerRebalanceListener;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.consumer.ConsumerRecords;import org.apache.kafka.clients.consumer.KafkaConsumer;import org.apache.kafka.clients.consumer.OffsetAndMetadata;import org.apache.kafka.common.TopicPartition;import org.apache.kafka.common.errors.RebalanceInProgressException;import org.apache.kafka.common.errors.WakeupException;/** The same behaviour without Spring: commit on revoke, never on lost. */public final class OrderEventsLoop implements Runnable { private final KafkaConsumer<String, String> consumer; private final Map<TopicPartition, OffsetAndMetadata> processed = new HashMap<>(); public OrderEventsLoop(KafkaConsumer<String, String> consumer) { this.consumer = consumer; } @Override public void run() { consumer.subscribe(List.of("orders.events"), new ConsumerRebalanceListener() { @Override public void onPartitionsRevoked(Collection<TopicPartition> partitions) { Map<TopicPartition, OffsetAndMetadata> handingOver = new HashMap<>(); for (TopicPartition partition : partitions) { OffsetAndMetadata next = processed.remove(partition); if (next != null) { handingOver.put(partition, next); } } if (!handingOver.isEmpty()) { consumer.commitSync(handingOver); } } @Override public void onPartitionsLost(Collection<TopicPartition> partitions) { partitions.forEach(processed::remove); } @Override public void onPartitionsAssigned(Collection<TopicPartition> partitions) { // Nothing to load: positions come from the committed offsets. } }); try { while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500)); for (ConsumerRecord<String, String> record : records) { handle(record); processed.put( new TopicPartition(record.topic(), record.partition()), new OffsetAndMetadata(record.offset() + 1)); } commitProcessed(); } } catch (WakeupException shutdown) { // consumer.wakeup() was called from another thread to stop the loop } finally { consumer.close(); } } private void commitProcessed() { if (processed.isEmpty()) { return; } try { consumer.commitSync(new HashMap<>(processed)); } catch (RebalanceInProgressException retryLater) { // Not fatal. The next poll() completes the rebalance; the revoke callback // commits what moves away and the next pass commits what stays. } catch (CommitFailedException noLongerOwner) { // Fatal for these offsets: the group moved on without this member. // The new owners will read these records again. processed.clear(); } } private void handle(ConsumerRecord<String, String> record) { // Idempotent business logic goes here. }}Why it is written this way
- The committed value is
record.offset() + 1: a commit stores the position of the next record to read, not an acknowledgement of one record. - The revoke callback commits only the partitions it was given. With cooperative rebalancing the member keeps the others and must not disturb them.
RebalanceInProgressExceptionis handled by doing nothing: the offsets stay in the map, the nextpoll()finishes the rebalance and they are committed then.CommitFailedExceptionis not retried. It means the records were processed but their offsets cannot be stored, so duplicates downstream are certain andhandlemust tolerate them.close()in thefinallyblock is what turns a shutdown into an immediate leave for a dynamic member instead of a 45 second wait for the session timeout.
Configuration
Defaults are from the Apache Kafka 4.4 consumer and broker configuration reference. The first four client settings exist under both protocols unless the entry says otherwise.
group.protocolDefault classic
- What it does
- Which group protocol the consumer uses:
classic(JoinGroup and SyncGroup, client-side assignors) orconsumer(KIP-848, broker-side assignment). - When to change it
- Leave it on
classicwith a 3.x client. With 4.x clients and brokers,consumergives incremental rebalances without a group-wide pause; a group can be converted with a rolling restart.
partition.assignment.strategyDefault RangeAssignor, CooperativeStickyAssignor
- What it does
- Assignors this member supports, in order of preference. The group uses one that all members list. Classic protocol only.
- When to change it
- Remove
RangeAssignorin one rolling restart to get cooperative rebalancing. Never change the list so that old and new members share no assignor.
session.timeout.msDefault 45000 (45 seconds)
- What it does
- How long the coordinator waits without a heartbeat before it removes the member. Classic protocol only; must lie between the broker's
group.min.session.timeout.ms(6 s) andgroup.max.session.timeout.ms(30 min). - When to change it
- Lower it for faster crash detection if your network and garbage-collection pauses allow. Raise it to cover a restart when you use static membership, knowing dead members are then found that much later.
heartbeat.interval.msDefault 3000 (3 seconds)
- What it does
- How often the background thread heartbeats. Also bounds how quickly a member hears that a rebalance has started. Classic protocol only.
- When to change it
- Keep it at no more than a third of the session timeout so that a couple of lost heartbeats do not expire the member.
max.poll.interval.msDefault 300000 (5 minutes)
- What it does
- Longest allowed gap between two
poll()calls before the client gives up its membership. Also sent to the coordinator as the rebalance timeout. - When to change it
- Size it from worst-case batch time:
max.poll.recordstimes the slowest handling time, with margin. Prefer loweringmax.poll.recordsover raising this, because a large value also lets one slow member stretch a rebalance.
max.poll.recordsDefault 500
- What it does
- Upper limit on records returned by one
poll(). It does not change how much is fetched from the broker. - When to change it
- Lower it when single records can be slow, so that a batch always finishes inside the poll interval.
group.instance.idDefault null
- What it does
- Makes the member static. Only one live member per id is allowed in the group.
- When to change it
- Set it to something stable per instance, such as a StatefulSet pod name, when restarts are frequent and short. Leave it unset when instances are replaced under new names or when fast failover matters more than avoiding rebalances.
group.consumer.session.timeout.ms (broker)Default 45000 (45 seconds)
- What it does
- Session timeout for members of groups on the consumer protocol. Set on the broker;
group.consumer.heartbeat.interval.ms(5 s) is its companion. - When to change it
- Change it on the broker for all such groups, or override it per group through the group configuration
consumer.session.timeout.ms.
group.initial.rebalance.delay.ms (broker)Default 3000 (3 seconds)
- What it does
- How long the coordinator waits for more members before the first rebalance of a new, empty group.
- When to change it
- Usually left alone. It is why a freshly started group takes a few seconds before any member receives partitions.
Hands-on exercise
Count what stops when the group changes
Compare an eager and a cooperative assignor on the same events, then see what static membership changes for a restart and for a real crash.
Steps · 0 of 6 done
Rebalance the same group four ways
Simulation · not a real clusterUse the add, remove and crash controls, and switch the assignor and static membership between runs.
What you should see
Range revokes all 6 partitions for a join that only needed to move 2. Cooperative-sticky revokes just the 2 that move, in a second round, while the other 4 keep flowing. A crash is only noticed after the session timeout in both cases. With static membership a quick restart moves nothing, but a consumer that stays down leaves its partitions unread, with lag growing, until the session timeout expires.
What can go wrong
Processing longer than the poll interval
- Symptom
- Rebalances at irregular intervals,
CommitFailedExceptionin the logs, the same records processed again and again, lag that never drains. - Cause
- A batch took longer than
max.poll.interval.ms, so the client left the group. The partitions moved, the new owner hit the same slow batch, and the cycle repeats. - Fix
- Lower
max.poll.recordsuntil the worst-case batch fits, put a timeout on the downstream call, and only then consider raisingmax.poll.interval.ms. Make the handler idempotent because some redelivery is unavoidable.
Switching to cooperative in one step
- Symptom
- During the rollout, new instances fail to join the group with an inconsistent group protocol error, or the group stalls until the old instances are gone.
- Cause
- The new instances list only
CooperativeStickyAssignorwhile the old ones list only an eager assignor, so there is no assignor all members support. - Fix
- Do two rolling restarts: first list both assignors, then remove the eager one. With the untouched default list a single restart that removes
RangeAssignoris enough.
Static membership with a long session timeout and no alert
- Symptom
- After a pod is evicted and not rescheduled, a few partitions stop moving for many minutes while the rest of the group looks healthy.
- Cause
- A static member that disappears is only removed after
session.timeout.ms, which had been raised to cover restarts. Until then nobody owns its partitions. - Fix
- Keep the session timeout just above a real restart time, alert on per-partition lag, and remove a permanently gone instance explicitly through the Admin API if you cannot wait.
Committing from onPartitionsLost
- Symptom
- Exceptions thrown out of
poll()during recovery from a timeout, sometimes followed by another rebalance. - Cause
- The default
onPartitionsLostcallsonPartitionsRevoked, which tries to commit for partitions the member no longer owns. - Fix
- Override
onPartitionsLostto release local state only. In Spring, implementConsumerAwareRebalanceListener, whose lost callback does nothing by default.
Shutting down without closing the consumer
- Symptom
- Every deploy leaves some partitions idle for about 45 seconds, then a rebalance.
- Cause
- The process was killed before
close()ran, so no leave request was sent and the coordinator had to wait for the session timeout. - Fix
- Let the listener container stop on shutdown and give the process a termination grace period longer than the slowest in-flight batch.
How to troubleshoot
The group seems to rebalance constantly
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --state
- How to read it
- A healthy classic group shows
Stable. SeeingPreparingRebalanceorCompletingRebalanceon repeated runs means it is cycling. The same output names the assignor in use and the member count.
Unsure whether members are leaving by timeout or by restart
Check
Consumer metrics `rebalance-rate-per-hour`, `last-rebalance-seconds-ago` and `failed-rebalance-total` (kafka.consumer:type=consumer-coordinator-metrics)
- How to read it
- Rebalances that line up with deploys are restarts. A steady rate with no deploys points at timeouts; confirm with
onPartitionsLostlog lines orCommitFailedException.
Suspected poll interval breaches
Check
Consumer metrics `time-between-poll-max` and `last-poll-seconds-ago` (kafka.consumer:type=consumer-metrics)
- How to read it
- If
time-between-poll-maxapproachesmax.poll.interval.ms, batches are too slow for the setting. The client also logs a warning that namesmax.poll.interval.mswhen it leaves the group for this reason.
Some partitions are not being read after a change
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --members --verbose
- How to read it
- Lists each member with its partitions. A member with none is surplus or still waiting for the second cooperative round. A partition that appears under no member is waiting for a static member's session to expire.
Rebalances take a long time to complete
Check
Consumer metrics `rebalance-latency-max` and `join-time-max` (kafka.consumer:type=consumer-coordinator-metrics)
- How to read it
- A join time close to the processing time of your slowest batch means the JoinGroup round is waiting for a member that is busy outside
poll(). Shorter batches shorten the rebalance.
Key takeaways
- A rebalance starts on a join, a leave, a missed session timeout, an exceeded poll interval, or a change of subscription or partition count. Each is noticed at a different speed.
- In the classic protocol the assignment is computed by a member, and the group waits for every member to return to
poll(), for up tomax.poll.interval.ms. - Eager assignors revoke every partition on every rebalance. Cooperative sticky revokes only the partitions that move, in a second round, and all members must support it before you rely on it.
- The session timeout finds dead processes, the poll interval finds stuck ones, and the heartbeat interval only sets how often the member checks in.
- Static membership trades rebalances on restart for slower detection of a member that never comes back.
- Commit in the revoke callback, never in the lost callback, and treat every rebalance as a possible source of duplicates.
group.protocol=consumermoves assignment to the broker and removes the group-wide pause. It is generally available since 4.0 but not the default in 4.x.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A group of 6 members on the default assignor list handles records that occasionally take 90 seconds each. max.poll.records is 500 and max.poll.interval.ms is 300000. Several times an hour the logs show CommitFailedException and the whole group stops for minutes.
Which change addresses the cause?
Scenario 2 of 3
Twelve instances run with partition.assignment.strategy set explicitly to RoundRobinAssignor. You want cooperative rebalancing before the next peak. Deploys are rolling, one instance at a time, and the group must keep working throughout.
How do you roll the change out?
Scenario 3 of 3
A service uses static membership with session.timeout.ms at 10 minutes so that slow restarts never rebalance. A node fails at 14:00 and its pod is not rescheduled. At 14:06 an alert fires: 3 of 24 partitions have growing lag while the other 21 are at zero.
What explains it, and what is the right response now?
References
References: Apache Kafka 4.4: Consumer configs ↗ · Apache Kafka 4.4: Consumer rebalance protocol (KIP-848 status, settings that no longer apply) ↗ · KafkaConsumer Javadoc: detecting consumer failures, commit exceptions ↗ · ConsumerRebalanceListener Javadoc: revoked, lost and assigned callbacks ↗ · Apache Kafka 4.4: Broker configs (group.* settings) ↗ · KIP-429: Kafka Consumer Incremental Rebalance Protocol ↗ · KIP-345: Static membership ↗ · KIP-62: Heartbeats from a background thread and max.poll.interval.ms ↗ · KIP-848: The Next Generation of the Consumer Rebalance Protocol ↗ · Spring for Apache Kafka 3.3: Rebalancing listeners ↗ · Spring for Apache Kafka 3.3: Message listener containers (static membership suffix) ↗ · Report a problem