Skip to content

BuildBreak lab 10 of 15

Consumer lag during high load

A flash sale makes orders arrive faster than the consumers can process them. Watch lag grow, try two fixes that do not work, then add real capacity and measure the drain time.

Intermediate30 min

Read first: Consumer lag · The consumer and its poll loop · Partition sizing and consumer parallelism · Throughput and latency · Monitoring Kafka: the signals that matter first

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
orders: 3 partitions, RF 3
Producers
storefront
Consumer groups
fulfilment-service (3)
Guided run
7 stages: build → break → observe → fix → retest

1

Business scenario

The storefront publishes an OrderPlaced event for every order to the topic orders, which has 3 partitions. The fulfilment service runs 3 instances in one consumer group; each record costs one database transaction. On a normal day 9 orders arrive per second and the group can process 15, so lag is zero. At 20:00 a flash sale starts and orders arrive at 30 per second for several minutes. Customers begin to ask why their order confirmation has not arrived. The on-call engineer scales the service from 3 to 4 pods and nothing changes.

2

Starting architecture

Drawing diagram…

The storefront producer writes OrderPlaced events keyed by order id to the topic orders, which has three partitions. Each partition is read by one instance of the fulfilment service, fulfilment-1 to fulfilment-3. Every instance writes each record to the orders database in its own transaction.

flowchart LR
  SF["Storefront producer"] -->|"OrderPlaced, key = order id"| T["Topic orders, 3 partitions"]
  T -->|"partition 0"| F1["fulfilment-1"]
  T -->|"partition 1"| F2["fulfilment-2"]
  T -->|"partition 2"| F3["fulfilment-3"]
  F1 -->|"one transaction per record"| DB[("orders database")]
  F2 -->|"one transaction per record"| DB
  F3 -->|"one transaction per record"| DB
Three partitions, three consumers, one database transaction per record. Each consumer handles 5 records per second in this simulation.

3

Expected behaviour

  • In normal traffic the group processes records as fast as they arrive: lag stays near zero and the oldest waiting record is less than a second old.
  • During a spike the group has enough spare capacity that lag stays small, or at least drains within a known time after the spike ends.
  • Lag is watched as time (how old is the oldest unprocessed record) as well as a number of records.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 7

Normal traffic

Do this

  1. Turn on Storefront publishing (9 orders per second) and press play.
  2. Read the "Lag as time" panel: arriving per second, processed per second, and the age of the oldest waiting record.

Look for

  • The group can process 15 records per second (3 consumers, 200 ms per record in this simulation) and 9 arrive: lag stays near zero.
  • "Oldest waiting record" stays below one second.

Not done yet. The simulator checks this for you:consumed >= 40 && lag("fulfilment-service") < 15

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

9 orders per second in normal traffic.

Lag as time (simulated)

simulated

Lag in records says how much is waiting. Lag as time says how stale the newest result is: the age of the oldest record this group has not committed yet.

Lag
0 records
Oldest waiting record
0 s
Arriving
0 /s
Processed
0 /s
Time to drain at these rates
no backlog
Busiest partition, share of new records
0 %
Last backlog
none finished yet

orders: per partition

partitionownerarriving /slagoldest waiting
orders-0fulfilment-1000 s
orders-1fulfilment-2000 s
orders-2fulfilment-3000 s

Simulated metrics

computed by the simulation
Orders published
0
Orders processed
0
Lag
0records
Oldest waiting record
0s
Arriving
0/s
Processed
0/s
Partitions
3
Consumers with a partition
3
Consumers with no partition
0
Removed for not polling
0
Failed commits
0
Processed twice
0
Rebalances
0

Consumer group fulfilment-service

stablegeneration 1lag 0

RangeAssignor (eager)commit after batchmax.poll.records 10

  • fulfilment-1runningprocessed 0
    orders-0
  • fulfilment-2runningprocessed 0
    orders-1
  • fulfilment-3runningprocessed 0
    orders-2
Partitions of group fulfilment-service: owner, committed offset and lag
PartitionOwnerCommittedLag
orders-0 fulfilment-1none0
orders-1 fulfilment-2none0
orders-2 fulfilment-3none0

Partition logs

simulated records
  • orders-0leader b1ISR 1,2,3HW 0end 0

    empty

    fulfilment-service: committed none, lag 0

  • orders-1leader b2ISR 2,3,1HW 0end 0

    empty

    fulfilment-service: committed none, lag 0

  • orders-2leader b3ISR 3,1,2HW 0end 0

    empty

    fulfilment-service: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders (3 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.