BuildBreak lab 10 of 15
Consumer lag during high load
A flash sale makes orders arrive faster than the consumers can process them. Watch lag grow, try two fixes that do not work, then add real capacity and measure the drain time.
Read first: Consumer lag · The consumer and its poll loop · Partition sizing and consumer parallelism · Throughput and latency · Monitoring Kafka: the signals that matter first
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- orders: 3 partitions, RF 3
- Producers
- storefront
- Consumer groups
- fulfilment-service (3)
- Guided run
- 7 stages: build → break → observe → fix → retest
1
Business scenario
The storefront publishes an OrderPlaced event for every order to the topic orders, which has 3 partitions. The fulfilment service runs 3 instances in one consumer group; each record costs one database transaction. On a normal day 9 orders arrive per second and the group can process 15, so lag is zero. At 20:00 a flash sale starts and orders arrive at 30 per second for several minutes. Customers begin to ask why their order confirmation has not arrived. The on-call engineer scales the service from 3 to 4 pods and nothing changes.
2
Starting architecture
Drawing diagram…
The storefront producer writes OrderPlaced events keyed by order id to the topic orders, which has three partitions. Each partition is read by one instance of the fulfilment service, fulfilment-1 to fulfilment-3. Every instance writes each record to the orders database in its own transaction.
flowchart LR
SF["Storefront producer"] -->|"OrderPlaced, key = order id"| T["Topic orders, 3 partitions"]
T -->|"partition 0"| F1["fulfilment-1"]
T -->|"partition 1"| F2["fulfilment-2"]
T -->|"partition 2"| F3["fulfilment-3"]
F1 -->|"one transaction per record"| DB[("orders database")]
F2 -->|"one transaction per record"| DB
F3 -->|"one transaction per record"| DB3
Expected behaviour
- In normal traffic the group processes records as fast as they arrive: lag stays near zero and the oldest waiting record is less than a second old.
- During a spike the group has enough spare capacity that lag stays small, or at least drains within a known time after the spike ends.
- Lag is watched as time (how old is the oldest unprocessed record) as well as a number of records.
4 · Build → break → observe → fix → retest
Guided run
Normal traffic
Do this
- Turn on Storefront publishing (9 orders per second) and press play.
- Read the "Lag as time" panel: arriving per second, processed per second, and the age of the oldest waiting record.
Look for
- The group can process 15 records per second (3 consumers, 200 ms per record in this simulation) and 9 arrive: lag stays near zero.
- "Oldest waiting record" stays below one second.
Not done yet. The simulator checks this for you:consumed >= 40 && lag("fulfilment-service") < 15
Simulator
Simulation · not a real clusterControls for this stage
9 orders per second in normal traffic.
Lag as time (simulated)
simulatedLag in records says how much is waiting. Lag as time says how stale the newest result is: the age of the oldest record this group has not committed yet.
- Lag
- 0 records
- Oldest waiting record
- 0 s
- Arriving
- 0 /s
- Processed
- 0 /s
- Time to drain at these rates
- no backlog
- Busiest partition, share of new records
- 0 %
- Last backlog
- none finished yet
orders: per partition
| partition | owner | arriving /s | lag | oldest waiting |
|---|---|---|---|---|
| orders-0 | fulfilment-1 | 0 | 0 | 0 s |
| orders-1 | fulfilment-2 | 0 | 0 | 0 s |
| orders-2 | fulfilment-3 | 0 | 0 | 0 s |
Simulated metrics
computed by the simulation- Orders published
- 0
- Orders processed
- 0
- Lag
- 0records
- Oldest waiting record
- 0s
- Arriving
- 0/s
- Processed
- 0/s
- Partitions
- 3
- Consumers with a partition
- 3
- Consumers with no partition
- 0
- Removed for not polling
- 0
- Failed commits
- 0
- Processed twice
- 0
- Rebalances
- 0
Consumer group fulfilment-service
stablegeneration 1lag 0RangeAssignor (eager)commit after batchmax.poll.records 10
- fulfilment-1runningprocessed 0orders-0
- fulfilment-2runningprocessed 0orders-1
- fulfilment-3runningprocessed 0orders-2
| Partition | Owner | Committed | Lag |
|---|---|---|---|
| orders-0 | fulfilment-1 | none | 0 |
| orders-1 | fulfilment-2 | none | 0 |
| orders-2 | fulfilment-3 | none | 0 |
Partition logs
simulated records- orders-0leader b1ISR 1,2,3HW 0end 0
empty
fulfilment-service: committed none, lag 0
- orders-1leader b2ISR 2,3,1HW 0end 0
empty
fulfilment-service: committed none, lag 0
- orders-2leader b3ISR 3,1,2HW 0end 0
empty
fulfilment-service: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders (3 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.