Path 4 · lesson 2 of 5
Consumer groups and partition assignment
How a group id turns many consumers into one subscriber, how partitions are shared out by each assignor, and why parallelism stops at the partition count.
What you will learn
- Predict which consumer reads which partition for a given group, partition count and assignor
- Decide whether two services should share a
group.idor use separate ones - Choose an assignor and explain what it does to balance and to movement during a rebalance
- Calculate how many consumers a group can use and spot idle ones with the CLI
- Choose between raising Spring
concurrencyand adding application instances
Before you start
The concept
A consumer group is a set of consumers that share one group.id. Kafka treats the group as a single logical subscriber: each record of a subscribed topic is delivered to one consumer of the group. It achieves this at the level of partitions, not records. Every partition of the topic is assigned to exactly one member of the group, and that member reads all of it.
Groups are independent of each other. A second group with a different group.id gets its own assignment and its own committed offsets, and so reads every record again at its own pace. Nothing is copied: both groups read the same log, each with its own positions. This gives you both classic messaging shapes from one topic. Consumers in the same group split the work like workers on a queue. Consumers in different groups each receive everything, like subscribers in publish and subscribe.
Same group.id
- Partitions are divided among the members
- Each record is processed by one member
- Adding a member spreads the same work over more processes
- Use for instances of the same service
Different group.id
- Each group is assigned all partitions
- Each record is processed once per group
- Adding a group adds a new, independent reader
- Use for different services reading the same events
One group, six partitions
Simulation · not a real clusterAdd consumers one at a time and watch the partitions being divided. Keep adding past six and see what the extra members receive.
Why it exists
Without groups, sharing a topic between several instances of a service is your problem. Either every instance reads everything and you deduplicate, or you hard-code which instance reads which partitions with assign and then handle the day an instance dies and its partitions go unread. Hard-coding also means a deploy that changes the instance count needs a configuration change everywhere.
Groups move that bookkeeping into Kafka. Membership is dynamic: when a member joins, leaves or stops responding, the partitions are redistributed among the members that remain. That redistribution is a rebalance, and Consumer rebalancing covers its triggers and costs in detail.
Assigning whole partitions, rather than dealing out individual records, is what keeps per-key ordering. Records with the same key go to the same partition, and that partition has one reader in the group at any time, so that reader sees the key's records in log order. A broker that handed single records to whichever consumer was free would be simpler to scale and would lose that property. The price is the cap described above: the partition count, chosen when the topic is created, is the upper limit on a group's parallelism.
How it works inside
The group coordinator
Every group is managed by one broker, its group coordinator. Groups are mapped to coordinators by group name: the name selects a partition of the internal topic __consumer_offsets, and the broker that leads that partition is the coordinator for the group. A consumer finds it by sending a FindCoordinator request to any broker. The coordinator tracks who the members are, receives their heartbeats, starts rebalances and stores the group's committed offsets. If that broker fails, leadership of the offsets partition moves and the new leader takes over the role after loading the group's state from the log.
Who computes the assignment
With the classic protocol, which is still the client default (group.protocol=classic), the coordinator does not decide who gets what. Each consumer sends the list of assignors it supports in partition.assignment.strategy. The coordinator picks one that every member supports and nominates one member as group leader. That member runs the assignor in its own JVM and sends the result back, and the coordinator distributes it. The assignment logic is therefore client code, which is why all members must agree on an assignor.
With the new consumer protocol from KIP-848 (group.protocol=consumer), generally available since Kafka 4.0 but not the client default, the assignment is computed on the broker. partition.assignment.strategy is not usable in that mode. The broker offers uniform (the default) and range, listed in the broker setting group.consumer.assignors, and a consumer may ask for one with group.remote.assignor.
What each assignor produces
| Assignor | How it assigns | Rebalance style | Result to expect |
|---|---|---|---|
RangeAssignor | Topic by topic. Partitions in numeric order, consumers in sorted order, each consumer gets a consecutive block. If it does not divide evenly, the first consumers get one extra | Eager | Partition N of every topic lands on the same consumer. With many topics, the first consumers collect all the extras |
RoundRobinAssignor | All partitions of all topics laid out together and dealt one at a time | Eager | Counts differ by at most one when all members subscribe to the same topics. Most partitions change owner when membership changes |
StickyAssignor | As balanced as possible first, then keeps as many existing assignments as it can | Eager | Balanced like round robin, with far fewer partitions changing owner |
CooperativeStickyAssignor | The same sticky logic | Cooperative | Same result as sticky, but members only give up the partitions that actually move |
The Javadoc example makes the difference concrete. Two consumers, C0 and C1, subscribe to topics t0 and t1 with 3 partitions each. Range gives C0 t0p0, t0p1, t1p0, t1p1 and C1 t0p2, t1p2: four against two, because C0 receives the extra partition of each topic. Round robin gives C0 t0p0, t0p2, t1p1 and C1 t0p1, t1p0, t1p2: three each.
Eager and cooperative differ in what happens while the group rebalances. With an eager assignor every member gives up all of its partitions before the new assignment arrives, so the whole group stops for a moment. With the cooperative one, members keep their partitions and only the ones changing owner are revoked, which takes a second, short rebalance round but leaves the rest of the group working.
Spring: concurrency is consumers, not a thread pool
In Spring for Apache Kafka, concurrency = 3 on a listener makes the ConcurrentMessageListenerContainer create three KafkaMessageListenerContainer instances. Each has its own KafkaConsumer and its own thread, and each joins the group as a separate member. From Kafka's side, one application instance with concurrency 3 looks exactly like three members. The total number of members is instances multiplied by concurrency, and the partition cap applies to that total.
The Spring reference gives a case worth remembering: three topics with five partitions each and concurrency=15. With the default range assignor only five consumers are active, each with one partition from every topic, and ten sit idle, because range works topic by topic and each topic has only five partitions to hand out. Round robin or a sticky assignor would spread the 15 partitions over all 15 consumers.
Partitions & consumer groups
Simulation, not a real clusterThe animated Build lesson shows records flowing through partitions into a group, step by step.
Step 1 · The pattern
A log and a reader
The producer appends messages to a partition: an ordered log. A consumer reads them one after another, in the order they were written.
- message
- consumed
Architecture
Drawing diagram…
Flowchart of a topic with six partitions. In the payment-service group, instance A reads partitions 0 and 1, instance B reads 2 and 3, and instance C reads 4 and 5. A second group, notification-service, has a single instance that reads all six partitions. All consumers send heartbeats and commits to a group coordinator broker.
flowchart LR
subgraph TOPIC["Topic orders.events with 6 partitions"]
P0["P0"]
P1["P1"]
P2["P2"]
P3["P3"]
P4["P4"]
P5["P5"]
end
subgraph PAY["Group payment-service"]
A["Instance A"]
B["Instance B"]
C["Instance C"]
end
subgraph NOTIF["Group notification-service"]
N["Single instance"]
end
GC["Group coordinator broker"]
P0 --> A
P1 --> A
P2 --> B
P3 --> B
P4 --> C
P5 --> C
TOPIC -->|"all 6 partitions"| N
A -.->|"heartbeats and commits"| GC
B -.-> GC
C -.-> GC
N -.-> GC- Inside
payment-serviceeach partition has exactly one reader, so each order event is handled by one payment instance. notification-serviceis a separate group, so it receives every event as well, with its own committed offsets. Its lag and its failures do not affect the payment group.- Both groups talk to a coordinator. It may be a different broker for each group, because the coordinator is chosen from the group name.
- A fourth, fifth or sixth payment instance would each take partitions from the others. A seventh would be assigned nothing.
Message flow
Drawing diagram…
Sequence diagram of a consumer joining an existing group under the classic protocol. Consumer B finds the coordinator and sends JoinGroup. Consumer A learns of the rebalance through a heartbeat response and rejoins. The coordinator names A as leader. A runs the assignor and sends the result with SyncGroup. The coordinator returns partitions 0 to 2 to A and 3 to 5 to B, and B fetches the committed offsets for its new partitions.
sequenceDiagram participant A as Consumer A participant B as Consumer B, new participant G as Group coordinator Note over A,G: A owns partitions 0 to 5 B->>G: FindCoordinator for payment-service B->>G: JoinGroup with supported assignors G-->>A: heartbeat response says rebalance in progress A->>G: JoinGroup again from inside poll G-->>A: you are the leader, members are A and B G-->>B: joined, wait for assignment A->>A: run assignor, A gets 0 to 2, B gets 3 to 5 A->>G: SyncGroup with the assignment B->>G: SyncGroup G-->>A: partitions 0, 1, 2 G-->>B: partitions 3, 4, 5 B->>G: fetch committed offsets for 3, 4, 5 G-->>B: resume positions Note over A,B: both consumers fetch from their partitions
- Existing members learn about a rebalance from a heartbeat response and rejoin from inside
poll. A member that is slow to callpolldelays the whole group. - The coordinator chooses the leader, and the leader's client code computes the assignment. That is why the assignor is a client setting in the classic protocol.
- B starts each new partition from the group's committed offset, not from wherever A's in-memory position was. Anything A processed but had not committed is read again by B.
- Under the new consumer protocol the JoinGroup and SyncGroup steps are replaced by heartbeats that carry the assignment, computed by the coordinator.
In a real system
A typical order pipeline publishes every order event to orders.events with 12 partitions, keyed by orderId. Three services react to those events, and each is its own group because each must see every event.
| Group id | Instances × concurrency | Members | Partitions per member | Why |
|---|---|---|---|---|
payment-service | 4 × 3 | 12 | 1 | Slow external calls, so it uses every partition |
inventory-service | 2 × 2 | 4 | 3 | Fast local writes, four readers are enough |
notification-service | 2 × 1 | 2 | 6 | Low volume. Two instances for availability |
When payments fall behind during a sale, the team's first instinct is to scale the Payment Service from 4 to 6 instances. That creates 18 members for 12 partitions: 6 of them are idle and the lag does not move. The real options are to make each record cheaper, to increase the partition count (which changes which partition each key maps to, see Keys and partitioning), or to process a partition's records concurrently inside the consumer while taking care of ordering and commits. The sizing question is treated in Partition sizing and parallelism.
Java and Spring Boot
Two listeners on the same topic in two different groups, with explicit concurrency, and the consumer settings that choose the assignor. In a real system the two listeners would live in separate services; they are shown together to make the group ids easy to compare.
package com.example.orders.events;import java.time.Instant;public record OrderPlaced(String orderId, String customerId, long amountMinor, String currency, Instant placedAt) {}Why it is written this way
- The event both groups read. Its key on the topic is
orderId, which is what keeps one order's events on one partition.
package com.example.payments;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.support.KafkaHeaders;import org.springframework.messaging.handler.annotation.Header;import org.springframework.messaging.handler.annotation.Payload;import org.springframework.stereotype.Component;import com.example.orders.events.OrderPlaced;@Componentpublic class PaymentOrderListener { private static final Logger log = LoggerFactory.getLogger(PaymentOrderListener.class); @KafkaListener( id = "payment-orders", topics = "orders.events", groupId = "payment-service", concurrency = "3") public void onOrderPlaced(@Payload OrderPlaced event, @Header(KafkaHeaders.RECEIVED_PARTITION) int partition, @Header(KafkaHeaders.OFFSET) long offset) { log.info("payment for order {} (partition {}, offset {}) on thread {}", event.orderId(), partition, offset, Thread.currentThread().getName()); // authorise the payment }}Why it is written this way
concurrency = "3"creates three consumers in this JVM. Each is a separate member ofpayment-servicewith its own thread and its own partitions.- Logging the thread name shows the mapping in practice: the same partition always appears on the same thread until the next rebalance.
- With 12 partitions and 4 instances this gives 12 members, one partition each. A fifth instance would add three idle members.
- The listener
idnames the container. The group comes fromgroupId; whengroupIdis absent, theidis used as the group id by default, which is an easy way to end up with an unintended group.
package com.example.notifications;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import com.example.orders.events.OrderPlaced;@Componentpublic class NotificationOrderListener { private final OrderEmailSender emails; public NotificationOrderListener(OrderEmailSender emails) { this.emails = emails; } @KafkaListener(id = "notification-orders", topics = "orders.events", groupId = "notification-service") public void onOrderPlaced(OrderPlaced event) { emails.sendOrderConfirmation(event.orderId(), event.customerId()); } /** Implemented elsewhere. */ public interface OrderEmailSender { void sendOrderConfirmation(String orderId, String customerId); }}Why it is written this way
- A different
groupIdmeans this listener receives every order event too, independently of the payment group. - No
concurrencyattribute: the container factory default applies, which is one consumer unlessspring.kafka.listener.concurrencysays otherwise.
spring: kafka: bootstrap-servers: localhost:9092 consumer: auto-offset-reset: earliest key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer properties: partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor spring.json.trusted.packages: com.example.orders.events spring.json.value.default.type: com.example.orders.events.OrderPlaced listener: concurrency: 1Why it is written this way
partition.assignment.strategyhas no dedicated Spring Boot property, so it goes underpropertieswith its Kafka name.CooperativeStickyAssignorkeeps assignments stable across deploys and avoids stopping the whole group on each rebalance. Set it from the first deploy of a new group. An existing group on range needs the two-step rolling change described in this lesson.spring.kafka.listener.concurrencyis the default for all listeners. Theconcurrencyattribute on a@KafkaListeneroverrides it for that listener.- The group id is set on each listener here, not globally, because the two listeners must be in different groups.
Configuration
Settings that decide group membership and assignment, with defaults documented for Kafka 4.x. The client settings are the same in the 3.8 and 3.9 clients used by Spring for Apache Kafka 3.3.
group.idDefault null
- What it does
- The name of the consumer group. Required when using
subscribeor Kafka-based offset storage. - When to change it
- One id per logical service, stable across deploys. Changing it creates a new group with no committed offsets, which then starts wherever
auto.offset.resetsays.
partition.assignment.strategyDefault RangeAssignor, CooperativeStickyAssignor
- What it does
- Classic protocol: the assignors this consumer supports, in order of preference. The first one common to all members is used, so the default behaves as range.
- When to change it
- Use
CooperativeStickyAssignorfor new groups. Keep range when partition N of several topics must be read by the same consumer. Not usable withgroup.protocol=consumer.
group.protocolDefault classic
- What it does
- Which group protocol the consumer uses:
classic, orconsumerfor the KIP-848 protocol with broker-side assignment. - When to change it
- The consumer protocol is production-ready on Kafka 4.0 and later brokers and clients. With Spring Boot 3.4 the managed client is 3.8 or 3.9, so check the client version before switching.
group.remote.assignorDefault null
- What it does
- Consumer protocol only: the name of the broker-side assignor to use. When unset, the broker uses the first entry of
group.consumer.assignors. - When to change it
- Leave unset to get
uniform. Setrangewhen you relied onRangeAssignorco-locating the same partition numbers.
group.consumer.assignors (broker)Default uniform,range
- What it does
- Consumer protocol only: the assignors the brokers offer. The first is the default for groups that do not ask for one.
- When to change it
- Rarely changed. The documented mapping from client-side assignors is range to
range, and round robin, sticky and cooperative sticky touniform.
group.instance.idDefault null
- What it does
- Makes the consumer a static member with a fixed identity. Only one member with a given id may be in the group.
- When to change it
- Set a stable, unique value per consumer to avoid a rebalance when an instance restarts quickly. With Spring concurrency above one, the container adds a
-nsuffix per consumer.
spring.kafka.listener.concurrencyDefault none (the container uses 1)
- What it does
- The number of consumers, each with its own thread, that every listener container creates in this application instance.
- When to change it
- Raise it to use more partitions per instance when the work is I/O bound. Keep instances times concurrency at or below the partition count.
Hands-on exercise
Predict the assignment, then check it
Use the consumer group simulation to see what each assignor produces for the same group, where idle consumers come from, and how much moves when membership changes.
Steps · 0 of 6 done
Assignors side by side
Simulation · not a real clusterChange the assignor, the number of consumers and the number of partitions. Predict the assignment before each change.
What you should see
Range with two consumers gives 0 to 2 and 3 to 5. With three it gives 0 and 1, 2 and 3, 4 and 5, so partitions 2 and 3 change hands between existing members. Cooperative sticky moves only the partitions of the consumer that left. With 8 consumers and 6 partitions two consumers are idle, and with 4 partitions four are idle.
What can go wrong
More consumers than partitions
- Symptom
- Scaling a service out does not reduce lag. Some instances log that they were assigned no partitions.
- Cause
- Instances times concurrency is larger than the partition count. The extra members are in the group but own nothing.
- Fix
- Count members against partitions before scaling. To go faster, reduce the cost per record or add partitions, knowing that adding partitions changes the key to partition mapping.
Two different services with the same group id
- Symptom
- Each service appears to process only a fraction of the events. No errors anywhere.
- Cause
- Both applications joined one group, so the partitions were split between them instead of each receiving all of them.
- Fix
- Give each service its own
group.id. Verify with--describe --members --verbosethat the member list of each group contains only that service's client ids.
Range assignor across many topics
- Symptom
- With one listener on several topics, a few consumers are busy and others are idle or nearly idle, even though there are enough partitions in total.
- Cause
- Range assigns each topic separately and always gives the extra partitions to the first consumers in sorted order. With topics that have fewer partitions than there are consumers, the later consumers get nothing.
- Fix
- Use
CooperativeStickyAssignor(or round robin) for multi-topic subscriptions, or split the topics across separate listeners with their own concurrency.
Changing the assignor in one deploy
- Symptom
- During a rolling deploy that switches from range to cooperative sticky, consumers fail to join with an inconsistent group protocol error.
- Cause
- Old and new members have no assignor in common, so the coordinator cannot choose one.
- Fix
- Roll once with both assignors listed, cooperative sticky first, range second, then roll again with range removed. This is what the default list is designed for.
How to troubleshoot
Need to see who owns what
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --members --verbose
- How to read it
- One line per member with its client id, host, number of partitions and the assignment. A member with 0 in the #PARTITIONS column is idle.
Not sure which assignor the group really uses
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --state
- How to read it
- Shows the coordinator broker, the assignment strategy in use, the group state and the member count.
rangehere on a group you meant to be sticky means the default list is still in effect.
A partition's lag grows while others stay flat
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
- How to read it
- Read the CONSUMER-ID column for that partition. Empty means nobody owns it. A consumer id means that member is slow or stuck, or the partition holds a hot key.
A service seems to miss events
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
- How to read it
- Confirm the group you expect exists and that there is one group per service. Then describe its members and look for hosts or client ids that belong to another application.
Key takeaways
- A group is one logical subscriber. Within a group each partition has exactly one reader.
- Different groups are independent: each gets every record and keeps its own offsets.
- A group cannot use more consumers than partitions. Extra members are idle standbys.
- The default assignor list behaves as range. Prefer cooperative sticky for new groups and change existing groups in two rolling steps.
- In the classic protocol a member computes the assignment. In the new consumer protocol the broker does, and the client assignor setting does not apply.
- Spring
concurrencycreates real consumers. Members equal instances times concurrency.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A topic has 12 partitions. The consuming service runs 4 pods with concurrency = 3 and lag is growing by about 2,000 records a minute. Each record takes 80 ms, almost all of it waiting for a database. Someone proposes scaling to 8 pods.
What will scaling to 8 pods do?
Scenario 2 of 3
One Spring listener subscribes to three topics with 5 partitions each and concurrency = 15. All consumer settings are defaults. Monitoring shows 5 busy consumer threads and 10 that never receive a record.
Which change puts all 15 consumers to work?
Scenario 3 of 3
A new fraud-check service must examine every order event that the payment service already consumes from orders.events. A developer copies the payment service's Kafka configuration, including group.id=payment-service, and deploys two instances.
What happens once the fraud service is running?
References
References: KafkaConsumer Javadoc (Kafka 4.4): consumer groups and topic subscriptions ↗ · Apache Kafka 4.4 documentation: Consumer Configs ↗ · RangeAssignor Javadoc (Kafka 4.4) ↗ · RoundRobinAssignor Javadoc (Kafka 4.4) ↗ · StickyAssignor Javadoc (Kafka 4.4) ↗ · CooperativeStickyAssignor Javadoc (Kafka 4.4) ↗ · Apache Kafka 4.4 documentation: Consumer Rebalance Protocol ↗ · KIP-848: The Next Generation of the Consumer Rebalance Protocol ↗ · Apache Kafka 4.4 documentation: Distribution (consumer offset tracking and the group coordinator) ↗ · Apache Kafka 4.4 documentation: Basic Kafka Operations (managing consumer groups) ↗ · Spring for Apache Kafka 3.3 reference: Message Listener Containers (ConcurrentMessageListenerContainer) ↗ · Spring for Apache Kafka 3.3 reference: @KafkaListener Annotation ↗ · Report a problem