BuildBreak lab 4 of 15
Adding and removing consumers
A rolling deploy pauses a whole consumer group, and a slow consumer is thrown out and its work redone. Fix both with three consumer settings.
Read first: Consumer rebalancing · The consumer and its poll loop
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- orders: 6 partitions, RF 3
- Producers
- order-service
- Consumer groups
- fulfilment (3)
- Guided run
- 6 stages: build → break → observe → fix → retest
1
Business scenario
The fulfilment service turns paid orders into warehouse pick jobs. Three instances share the six partitions of orders, and in steady state they keep up easily. Two things keep going wrong. Every deploy, a routine rolling restart, is followed by a burst of late pick jobs. And on days when the warehouse API is slow for one instance, that instance keeps disappearing from the group and reappearing, and the warehouse receives the same pick job twice. Both are rebalances: one the team causes, one the group coordinator forces.
2
Starting architecture
Drawing diagram…
The order service producer writes to the orders topic with six partitions. Three fulfilment consumers each own two partitions and call the warehouse API. The group coordinator tracks the members through heartbeats and hands out assignments.
flowchart LR
OS["Order service producer"] --> T["Topic orders with 6 partitions"]
subgraph G["Consumer group fulfilment"]
F1["fulfilment-1 owns P0 and P1"]
F2["fulfilment-2 owns P2 and P3"]
F3["fulfilment-3 owns P4 and P5"]
end
T --> F1
T --> F2
T --> F3
GC["Group coordinator"] -.->|"heartbeats and assignments"| G
F1 --> WH["Warehouse API"]
F2 --> WH
F3 --> WH3
Expected behaviour
- Restarting one instance affects only the partitions that instance owns, and only for as long as it is down.
- A consumer that is slow stays in the group and keeps its partitions; its records are late, not repeated.
- No pick job is created twice because of a membership change.
4 · Build → break → observe → fix → retest
Guided run
Steady state
Do this
- Turn on Order service publishing (10 orders per second) and press play.
- Read the fulfilment group panel: three members, two partitions each.
Look for
- "Rebalances" is 0 and the lag stays close to 0.
- The group uses Kafka's default assignor list, which behaves as the range assignor with the eager protocol.
Not done yet. The simulator checks this for you:consumed >= 40 && rebalances == 0
Simulator
Simulation · not a real clusterControls for this stage
10 orders per second across six partitions.
Consumer group fulfilment
stablegeneration 1lag 0RangeAssignor (eager)commit after batchmax.poll.records 50
- fulfilment-1runningprocessed 0orders-0orders-1
- fulfilment-2runningprocessed 0orders-2orders-3
- fulfilment-3runningprocessed 0orders-4orders-5
| Partition | Owner | Committed | Lag |
|---|---|---|---|
| orders-0 | fulfilment-1 | none | 0 |
| orders-1 | fulfilment-1 | none | 0 |
| orders-2 | fulfilment-2 | none | 0 |
| orders-3 | fulfilment-2 | none | 0 |
| orders-4 | fulfilment-3 | none | 0 |
| orders-5 | fulfilment-3 | none | 0 |
Simulated metrics
computed by the simulation- Rebalances
- 0
- Whole group paused
- 0s
- Partitions revoked
- 0
- Instance restarts
- 0
- Members removed for slow polling
- 0
- Rejected commits
- 0
- Duplicates processed
- 0
- Group lag
- 0records
Partition logs
simulated records- orders-0leader b1ISR 1,2,3HW 0end 0
empty
fulfilment: committed none, lag 0
- orders-1leader b2ISR 2,3,1HW 0end 0
empty
fulfilment: committed none, lag 0
- orders-2leader b3ISR 3,1,2HW 0end 0
empty
fulfilment: committed none, lag 0
- orders-3leader b1ISR 1,2,3HW 0end 0
empty
fulfilment: committed none, lag 0
- orders-4leader b2ISR 2,3,1HW 0end 0
empty
fulfilment: committed none, lag 0
- orders-5leader b3ISR 3,1,2HW 0end 0
empty
fulfilment: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders (6 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.