Skip to content

BuildBreak lab 4 of 15

Adding and removing consumers

A rolling deploy pauses a whole consumer group, and a slow consumer is thrown out and its work redone. Fix both with three consumer settings.

Intermediate25 min

Read first: Consumer rebalancing · The consumer and its poll loop

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
orders: 6 partitions, RF 3
Producers
order-service
Consumer groups
fulfilment (3)
Guided run
6 stages: build → break → observe → fix → retest

1

Business scenario

The fulfilment service turns paid orders into warehouse pick jobs. Three instances share the six partitions of orders, and in steady state they keep up easily. Two things keep going wrong. Every deploy, a routine rolling restart, is followed by a burst of late pick jobs. And on days when the warehouse API is slow for one instance, that instance keeps disappearing from the group and reappearing, and the warehouse receives the same pick job twice. Both are rebalances: one the team causes, one the group coordinator forces.

2

Starting architecture

Drawing diagram…

The order service producer writes to the orders topic with six partitions. Three fulfilment consumers each own two partitions and call the warehouse API. The group coordinator tracks the members through heartbeats and hands out assignments.

flowchart LR
  OS["Order service producer"] --> T["Topic orders with 6 partitions"]
  subgraph G["Consumer group fulfilment"]
    F1["fulfilment-1 owns P0 and P1"]
    F2["fulfilment-2 owns P2 and P3"]
    F3["fulfilment-3 owns P4 and P5"]
  end
  T --> F1
  T --> F2
  T --> F3
  GC["Group coordinator"] -.->|"heartbeats and assignments"| G
  F1 --> WH["Warehouse API"]
  F2 --> WH
  F3 --> WH
Three fulfilment instances share six partitions. The group coordinator tracks membership through heartbeats and poll calls.

3

Expected behaviour

  • Restarting one instance affects only the partitions that instance owns, and only for as long as it is down.
  • A consumer that is slow stays in the group and keeps its partitions; its records are late, not repeated.
  • No pick job is created twice because of a membership change.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 6

Steady state

Do this

  1. Turn on Order service publishing (10 orders per second) and press play.
  2. Read the fulfilment group panel: three members, two partitions each.

Look for

  • "Rebalances" is 0 and the lag stays close to 0.
  • The group uses Kafka's default assignor list, which behaves as the range assignor with the eager protocol.

Not done yet. The simulator checks this for you:consumed >= 40 && rebalances == 0

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

10 orders per second across six partitions.

Consumer group fulfilment

stablegeneration 1lag 0

RangeAssignor (eager)commit after batchmax.poll.records 50

  • fulfilment-1runningprocessed 0
    orders-0orders-1
  • fulfilment-2runningprocessed 0
    orders-2orders-3
  • fulfilment-3runningprocessed 0
    orders-4orders-5
Partitions of group fulfilment: owner, committed offset and lag
PartitionOwnerCommittedLag
orders-0 fulfilment-1none0
orders-1 fulfilment-1none0
orders-2 fulfilment-2none0
orders-3 fulfilment-2none0
orders-4 fulfilment-3none0
orders-5 fulfilment-3none0

Simulated metrics

computed by the simulation
Rebalances
0
Whole group paused
0s
Partitions revoked
0
Instance restarts
0
Members removed for slow polling
0
Rejected commits
0
Duplicates processed
0
Group lag
0records

Partition logs

simulated records
  • orders-0leader b1ISR 1,2,3HW 0end 0

    empty

    fulfilment: committed none, lag 0

  • orders-1leader b2ISR 2,3,1HW 0end 0

    empty

    fulfilment: committed none, lag 0

  • orders-2leader b3ISR 3,1,2HW 0end 0

    empty

    fulfilment: committed none, lag 0

  • orders-3leader b1ISR 1,2,3HW 0end 0

    empty

    fulfilment: committed none, lag 0

  • orders-4leader b2ISR 2,3,1HW 0end 0

    empty

    fulfilment: committed none, lag 0

  • orders-5leader b3ISR 3,1,2HW 0end 0

    empty

    fulfilment: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders (6 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.