Skip to content

Path 4 · lesson 2 of 5

Intermediate24 minApache Kafka 4.x (defaults checked against the 4.0 and 4.4 docs) · Java 21 · Spring Boot 3.4 · Spring for Apache Kafka 3.3 (kafka-clients 3.8 to 3.9)

Consumer groups and partition assignment

How a group id turns many consumers into one subscriber, how partitions are shared out by each assignor, and why parallelism stops at the partition count.

What you will learn

  • Predict which consumer reads which partition for a given group, partition count and assignor
  • Decide whether two services should share a group.id or use separate ones
  • Choose an assignor and explain what it does to balance and to movement during a rebalance
  • Calculate how many consumers a group can use and spot idle ones with the CLI
  • Choose between raising Spring concurrency and adding application instances

Before you start

The concept

A consumer group is a set of consumers that share one group.id. Kafka treats the group as a single logical subscriber: each record of a subscribed topic is delivered to one consumer of the group. It achieves this at the level of partitions, not records. Every partition of the topic is assigned to exactly one member of the group, and that member reads all of it.

Groups are independent of each other. A second group with a different group.id gets its own assignment and its own committed offsets, and so reads every record again at its own pace. Nothing is copied: both groups read the same log, each with its own positions. This gives you both classic messaging shapes from one topic. Consumers in the same group split the work like workers on a queue. Consumers in different groups each receive everything, like subscribers in publish and subscribe.

Same group.id

  • Partitions are divided among the members
  • Each record is processed by one member
  • Adding a member spreads the same work over more processes
  • Use for instances of the same service

Different group.id

  • Each group is assigned all partitions
  • Each record is processed once per group
  • Adding a group adds a new, independent reader
  • Use for different services reading the same events

One group, six partitions

Simulation · not a real cluster

Add consumers one at a time and watch the partitions being divided. Keep adding past six and see what the extra members receive.

Why it exists

Without groups, sharing a topic between several instances of a service is your problem. Either every instance reads everything and you deduplicate, or you hard-code which instance reads which partitions with assign and then handle the day an instance dies and its partitions go unread. Hard-coding also means a deploy that changes the instance count needs a configuration change everywhere.

Groups move that bookkeeping into Kafka. Membership is dynamic: when a member joins, leaves or stops responding, the partitions are redistributed among the members that remain. That redistribution is a rebalance, and Consumer rebalancing covers its triggers and costs in detail.

Assigning whole partitions, rather than dealing out individual records, is what keeps per-key ordering. Records with the same key go to the same partition, and that partition has one reader in the group at any time, so that reader sees the key's records in log order. A broker that handed single records to whichever consumer was free would be simpler to scale and would lose that property. The price is the cap described above: the partition count, chosen when the topic is created, is the upper limit on a group's parallelism.

How it works inside

The group coordinator

Every group is managed by one broker, its group coordinator. Groups are mapped to coordinators by group name: the name selects a partition of the internal topic __consumer_offsets, and the broker that leads that partition is the coordinator for the group. A consumer finds it by sending a FindCoordinator request to any broker. The coordinator tracks who the members are, receives their heartbeats, starts rebalances and stores the group's committed offsets. If that broker fails, leadership of the offsets partition moves and the new leader takes over the role after loading the group's state from the log.

Who computes the assignment

With the classic protocol, which is still the client default (group.protocol=classic), the coordinator does not decide who gets what. Each consumer sends the list of assignors it supports in partition.assignment.strategy. The coordinator picks one that every member supports and nominates one member as group leader. That member runs the assignor in its own JVM and sends the result back, and the coordinator distributes it. The assignment logic is therefore client code, which is why all members must agree on an assignor.

With the new consumer protocol from KIP-848 (group.protocol=consumer), generally available since Kafka 4.0 but not the client default, the assignment is computed on the broker. partition.assignment.strategy is not usable in that mode. The broker offers uniform (the default) and range, listed in the broker setting group.consumer.assignors, and a consumer may ask for one with group.remote.assignor.

What each assignor produces

AssignorHow it assignsRebalance styleResult to expect
RangeAssignorTopic by topic. Partitions in numeric order, consumers in sorted order, each consumer gets a consecutive block. If it does not divide evenly, the first consumers get one extraEagerPartition N of every topic lands on the same consumer. With many topics, the first consumers collect all the extras
RoundRobinAssignorAll partitions of all topics laid out together and dealt one at a timeEagerCounts differ by at most one when all members subscribe to the same topics. Most partitions change owner when membership changes
StickyAssignorAs balanced as possible first, then keeps as many existing assignments as it canEagerBalanced like round robin, with far fewer partitions changing owner
CooperativeStickyAssignorThe same sticky logicCooperativeSame result as sticky, but members only give up the partitions that actually move
The four built-in client-side assignors. Eager and cooperative describe what is revoked during a rebalance.

The Javadoc example makes the difference concrete. Two consumers, C0 and C1, subscribe to topics t0 and t1 with 3 partitions each. Range gives C0 t0p0, t0p1, t1p0, t1p1 and C1 t0p2, t1p2: four against two, because C0 receives the extra partition of each topic. Round robin gives C0 t0p0, t0p2, t1p1 and C1 t0p1, t1p0, t1p2: three each.

Eager and cooperative differ in what happens while the group rebalances. With an eager assignor every member gives up all of its partitions before the new assignment arrives, so the whole group stops for a moment. With the cooperative one, members keep their partitions and only the ones changing owner are revoked, which takes a second, short rebalance round but leaves the rest of the group working.

Spring: concurrency is consumers, not a thread pool

In Spring for Apache Kafka, concurrency = 3 on a listener makes the ConcurrentMessageListenerContainer create three KafkaMessageListenerContainer instances. Each has its own KafkaConsumer and its own thread, and each joins the group as a separate member. From Kafka's side, one application instance with concurrency 3 looks exactly like three members. The total number of members is instances multiplied by concurrency, and the partition cap applies to that total.

The Spring reference gives a case worth remembering: three topics with five partitions each and concurrency=15. With the default range assignor only five consumers are active, each with one partition from every topic, and ten sit idle, because range works topic by topic and each topic has only five partitions to hand out. Round robin or a sticky assignor would spread the 15 partitions over all 15 consumers.

Partitions & consumer groups

Simulation, not a real cluster

The animated Build lesson shows records flowing through partitions into a group, step by step.

Step 1 · The pattern

A log and a reader

The producer appends messages to a partition: an ordered log. A consumer reads them one after another, in the order they were written.

  • message
  • consumed
Open the full animated lesson

Architecture

Drawing diagram…

Flowchart of a topic with six partitions. In the payment-service group, instance A reads partitions 0 and 1, instance B reads 2 and 3, and instance C reads 4 and 5. A second group, notification-service, has a single instance that reads all six partitions. All consumers send heartbeats and commits to a group coordinator broker.

flowchart LR
  subgraph TOPIC["Topic orders.events with 6 partitions"]
    P0["P0"]
    P1["P1"]
    P2["P2"]
    P3["P3"]
    P4["P4"]
    P5["P5"]
  end
  subgraph PAY["Group payment-service"]
    A["Instance A"]
    B["Instance B"]
    C["Instance C"]
  end
  subgraph NOTIF["Group notification-service"]
    N["Single instance"]
  end
  GC["Group coordinator broker"]
  P0 --> A
  P1 --> A
  P2 --> B
  P3 --> B
  P4 --> C
  P5 --> C
  TOPIC -->|"all 6 partitions"| N
  A -.->|"heartbeats and commits"| GC
  B -.-> GC
  C -.-> GC
  N -.-> GC
Two groups on one topic: the payment group splits six partitions three ways, the notification group reads all six with one consumer.
  • Inside payment-service each partition has exactly one reader, so each order event is handled by one payment instance.
  • notification-service is a separate group, so it receives every event as well, with its own committed offsets. Its lag and its failures do not affect the payment group.
  • Both groups talk to a coordinator. It may be a different broker for each group, because the coordinator is chosen from the group name.
  • A fourth, fifth or sixth payment instance would each take partitions from the others. A seventh would be assigned nothing.

Message flow

Drawing diagram…

Sequence diagram of a consumer joining an existing group under the classic protocol. Consumer B finds the coordinator and sends JoinGroup. Consumer A learns of the rebalance through a heartbeat response and rejoins. The coordinator names A as leader. A runs the assignor and sends the result with SyncGroup. The coordinator returns partitions 0 to 2 to A and 3 to 5 to B, and B fetches the committed offsets for its new partitions.

sequenceDiagram
  participant A as Consumer A
  participant B as Consumer B, new
  participant G as Group coordinator
  Note over A,G: A owns partitions 0 to 5
  B->>G: FindCoordinator for payment-service
  B->>G: JoinGroup with supported assignors
  G-->>A: heartbeat response says rebalance in progress
  A->>G: JoinGroup again from inside poll
  G-->>A: you are the leader, members are A and B
  G-->>B: joined, wait for assignment
  A->>A: run assignor, A gets 0 to 2, B gets 3 to 5
  A->>G: SyncGroup with the assignment
  B->>G: SyncGroup
  G-->>A: partitions 0, 1, 2
  G-->>B: partitions 3, 4, 5
  B->>G: fetch committed offsets for 3, 4, 5
  G-->>B: resume positions
  Note over A,B: both consumers fetch from their partitions
A second consumer joins a classic-protocol group: the coordinator gathers the members, one member computes the assignment, the coordinator hands it out.
  1. Existing members learn about a rebalance from a heartbeat response and rejoin from inside poll. A member that is slow to call poll delays the whole group.
  2. The coordinator chooses the leader, and the leader's client code computes the assignment. That is why the assignor is a client setting in the classic protocol.
  3. B starts each new partition from the group's committed offset, not from wherever A's in-memory position was. Anything A processed but had not committed is read again by B.
  4. Under the new consumer protocol the JoinGroup and SyncGroup steps are replaced by heartbeats that carry the assignment, computed by the coordinator.

In a real system

A typical order pipeline publishes every order event to orders.events with 12 partitions, keyed by orderId. Three services react to those events, and each is its own group because each must see every event.

Group idInstances × concurrencyMembersPartitions per memberWhy
payment-service4 × 3121Slow external calls, so it uses every partition
inventory-service2 × 243Fast local writes, four readers are enough
notification-service2 × 126Low volume. Two instances for availability
Three independent groups on the same 12-partition topic.

When payments fall behind during a sale, the team's first instinct is to scale the Payment Service from 4 to 6 instances. That creates 18 members for 12 partitions: 6 of them are idle and the lag does not move. The real options are to make each record cheaper, to increase the partition count (which changes which partition each key maps to, see Keys and partitioning), or to process a partition's records concurrently inside the consumer while taking care of ordering and commits. The sizing question is treated in Partition sizing and parallelism.

Java and Spring Boot

Two listeners on the same topic in two different groups, with explicit concurrency, and the consumer settings that choose the assignor. In a real system the two listeners would live in separate services; they are shown together to make the group ids easy to compare.

OrderPlaced.javaJava
package com.example.orders.events;import java.time.Instant;public record OrderPlaced(String orderId, String customerId, long amountMinor, String currency, Instant placedAt) {}

Why it is written this way

  • The event both groups read. Its key on the topic is orderId, which is what keeps one order's events on one partition.

Configuration

Settings that decide group membership and assignment, with defaults documented for Kafka 4.x. The client settings are the same in the 3.8 and 3.9 clients used by Spring for Apache Kafka 3.3.

group.idDefault null
What it does
The name of the consumer group. Required when using subscribe or Kafka-based offset storage.
When to change it
One id per logical service, stable across deploys. Changing it creates a new group with no committed offsets, which then starts wherever auto.offset.reset says.
partition.assignment.strategyDefault RangeAssignor, CooperativeStickyAssignor
What it does
Classic protocol: the assignors this consumer supports, in order of preference. The first one common to all members is used, so the default behaves as range.
When to change it
Use CooperativeStickyAssignor for new groups. Keep range when partition N of several topics must be read by the same consumer. Not usable with group.protocol=consumer.
group.protocolDefault classic
What it does
Which group protocol the consumer uses: classic, or consumer for the KIP-848 protocol with broker-side assignment.
When to change it
The consumer protocol is production-ready on Kafka 4.0 and later brokers and clients. With Spring Boot 3.4 the managed client is 3.8 or 3.9, so check the client version before switching.
group.remote.assignorDefault null
What it does
Consumer protocol only: the name of the broker-side assignor to use. When unset, the broker uses the first entry of group.consumer.assignors.
When to change it
Leave unset to get uniform. Set range when you relied on RangeAssignor co-locating the same partition numbers.
group.consumer.assignors (broker)Default uniform,range
What it does
Consumer protocol only: the assignors the brokers offer. The first is the default for groups that do not ask for one.
When to change it
Rarely changed. The documented mapping from client-side assignors is range to range, and round robin, sticky and cooperative sticky to uniform.
group.instance.idDefault null
What it does
Makes the consumer a static member with a fixed identity. Only one member with a given id may be in the group.
When to change it
Set a stable, unique value per consumer to avoid a rebalance when an instance restarts quickly. With Spring concurrency above one, the container adds a -n suffix per consumer.
spring.kafka.listener.concurrencyDefault none (the container uses 1)
What it does
The number of consumers, each with its own thread, that every listener container creates in this application instance.
When to change it
Raise it to use more partitions per instance when the work is I/O bound. Keep instances times concurrency at or below the partition count.

Hands-on exercise

Predict the assignment, then check it

Use the consumer group simulation to see what each assignor produces for the same group, where idle consumers come from, and how much moves when membership changes.

Steps · 0 of 6 done

Assignors side by side

Simulation · not a real cluster

Change the assignor, the number of consumers and the number of partitions. Predict the assignment before each change.

Take it further in the lab: Consumer group assignment

What can go wrong

More consumers than partitions
Symptom
Scaling a service out does not reduce lag. Some instances log that they were assigned no partitions.
Cause
Instances times concurrency is larger than the partition count. The extra members are in the group but own nothing.
Fix
Count members against partitions before scaling. To go faster, reduce the cost per record or add partitions, knowing that adding partitions changes the key to partition mapping.
Two different services with the same group id
Symptom
Each service appears to process only a fraction of the events. No errors anywhere.
Cause
Both applications joined one group, so the partitions were split between them instead of each receiving all of them.
Fix
Give each service its own group.id. Verify with --describe --members --verbose that the member list of each group contains only that service's client ids.
Range assignor across many topics
Symptom
With one listener on several topics, a few consumers are busy and others are idle or nearly idle, even though there are enough partitions in total.
Cause
Range assigns each topic separately and always gives the extra partitions to the first consumers in sorted order. With topics that have fewer partitions than there are consumers, the later consumers get nothing.
Fix
Use CooperativeStickyAssignor (or round robin) for multi-topic subscriptions, or split the topics across separate listeners with their own concurrency.
Changing the assignor in one deploy
Symptom
During a rolling deploy that switches from range to cooperative sticky, consumers fail to join with an inconsistent group protocol error.
Cause
Old and new members have no assignor in common, so the coordinator cannot choose one.
Fix
Roll once with both assignors listed, cooperative sticky first, range second, then roll again with range removed. This is what the default list is designed for.

How to troubleshoot

Need to see who owns what

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --members --verbose
How to read it
One line per member with its client id, host, number of partitions and the assignment. A member with 0 in the #PARTITIONS column is idle.
Not sure which assignor the group really uses

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --state
How to read it
Shows the coordinator broker, the assignment strategy in use, the group state and the member count. range here on a group you meant to be sticky means the default list is still in effect.
A partition's lag grows while others stay flat

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
Read the CONSUMER-ID column for that partition. Empty means nobody owns it. A consumer id means that member is slow or stuck, or the partition holds a hot key.
A service seems to miss events

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
How to read it
Confirm the group you expect exists and that there is one group per service. Then describe its members and look for hosts or client ids that belong to another application.

Key takeaways

  • A group is one logical subscriber. Within a group each partition has exactly one reader.
  • Different groups are independent: each gets every record and keeps its own offsets.
  • A group cannot use more consumers than partitions. Extra members are idle standbys.
  • The default assignor list behaves as range. Prefer cooperative sticky for new groups and change existing groups in two rolling steps.
  • In the classic protocol a member computes the assignment. In the new consumer protocol the broker does, and the client assignor setting does not apply.
  • Spring concurrency creates real consumers. Members equal instances times concurrency.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A topic has 12 partitions. The consuming service runs 4 pods with concurrency = 3 and lag is growing by about 2,000 records a minute. Each record takes 80 ms, almost all of it waiting for a database. Someone proposes scaling to 8 pods.

What will scaling to 8 pods do?

Scenario 2 of 3

One Spring listener subscribes to three topics with 5 partitions each and concurrency = 15. All consumer settings are defaults. Monitoring shows 5 busy consumer threads and 10 that never receive a record.

Which change puts all 15 consumers to work?

Scenario 3 of 3

A new fraud-check service must examine every order event that the payment service already consumes from orders.events. A developer copies the payment service's Kafka configuration, including group.id=payment-service, and deploys two instances.

What happens once the fraud service is running?

References

References: KafkaConsumer Javadoc (Kafka 4.4): consumer groups and topic subscriptions ↗ · Apache Kafka 4.4 documentation: Consumer Configs ↗ · RangeAssignor Javadoc (Kafka 4.4) ↗ · RoundRobinAssignor Javadoc (Kafka 4.4) ↗ · StickyAssignor Javadoc (Kafka 4.4) ↗ · CooperativeStickyAssignor Javadoc (Kafka 4.4) ↗ · Apache Kafka 4.4 documentation: Consumer Rebalance Protocol ↗ · KIP-848: The Next Generation of the Consumer Rebalance Protocol ↗ · Apache Kafka 4.4 documentation: Distribution (consumer offset tracking and the group coordinator) ↗ · Apache Kafka 4.4 documentation: Basic Kafka Operations (managing consumer groups) ↗ · Spring for Apache Kafka 3.3 reference: Message Listener Containers (ConcurrentMessageListenerContainer) ↗ · Spring for Apache Kafka 3.3 reference: @KafkaListener Annotation ↗ · Report a problem