BuildBreak lab 11 of 15
Hot partition
One large merchant sends most of the traffic under a single key. One partition and one consumer are saturated while the rest idle. Find out why scaling does not help and what each fix costs.
Read first: Hot partitions and backpressure · Keys and partitioning: who decides where a record goes · Partition sizing and consumer parallelism · Consumer lag
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- merchant-orders: 4 partitions, RF 3 · merchant-orders-acme: 4 partitions, RF 3
- Producers
- small-merchants, acme, acme-dedicated
- Consumer groups
- order-projection (4), order-projection-acme (2)
- Guided run
- 8 stages: build → break → fix → retest
1
Business scenario
A commerce platform publishes every order event of every merchant to the topic merchant-orders, keyed by merchant id so that each merchant's events stay in order. The order projection service (4 instances, one per partition) builds the order pages that merchants see. With a dozen small merchants the load is even. Then Acme, a retailer larger than all the others together, goes live. Acme's order pages fall minutes behind, and so do the pages of two small merchants that have nothing to do with Acme. The dashboard shows the group at 40 percent of its capacity.
2
Starting architecture
Drawing diagram…
Twelve small merchants and one large merchant, Acme, publish to the topic merchant-orders with the merchant id as the record key. The topic has four partitions, each read by one instance of the order projection service, projection-1 to projection-4. All Acme records carry the key merchant-acme.
flowchart LR SM["12 small merchants"] -->|"key = merchant id"| T["Topic merchant-orders, 4 partitions"] AC["Acme, most of the traffic"] -->|"key = merchant-acme"| T T --> P0["partition 0"] T --> P1["partition 1"] T --> P2["partition 2"] T --> P3["partition 3"] P0 --> C1["projection-1"] P1 --> C2["projection-2"] P2 --> C3["projection-3"] P3 --> C4["projection-4"]
3
Expected behaviour
- Load is spread over the partitions evenly enough that no single consumer is the limit.
- The events of one order (created, paid, shipped) are processed in that order.
- A large merchant does not delay the small merchants that happen to share its partition.
4 · Build → break → observe → fix → retest
Guided run
A dozen small merchants
Do this
- Turn on Small merchants publishing (12 events per second across 12 merchants) and press play.
- Look at the per-partition table in the "Lag as time" panel.
Look for
- Each partition receives a few records per second; each consumer can process 10 per second. Lag is near zero everywhere.
- "Busiest partition, share of new records" is not far from 25 percent: an even spread over 4 partitions.
Not done yet. The simulator checks this for you:consumed >= 40 && lagAge("order-projection") < 2
Simulator
Simulation · not a real clusterControls for this stage
12 events per second, spread over 12 merchants.
Lag as time (simulated)
simulatedLag in records says how much is waiting. Lag as time says how stale the newest result is: the age of the oldest record this group has not committed yet.
- Lag
- 0 records
- Oldest waiting record
- 0 s
- Arriving
- 0 /s
- Processed
- 0 /s
- Time to drain at these rates
- no backlog
- Busiest partition, share of new records
- 0 %
- Last backlog
- none finished yet
merchant-orders: per partition
| partition | owner | arriving /s | lag | oldest waiting |
|---|---|---|---|---|
| merchant-orders-0 | projection-1 | 0 | 0 | 0 s |
| merchant-orders-1 | projection-2 | 0 | 0 | 0 s |
| merchant-orders-2 | projection-3 | 0 | 0 | 0 s |
| merchant-orders-3 | projection-4 | 0 | 0 | 0 s |
Simulated metrics
computed by the simulation- Records published
- 0
- Records processed
- 0
- Lag, shared topic
- 0records
- Largest partition lag
- 0records
- Oldest waiting record
- 0s
- Busiest partition, share of new records
- 0%
- Processed, shared topic
- 0/s
- Partitions
- 4
- Consumers with no partition
- 0
- Events processed out of order
- 0
- Lag, Acme's own topic
- 0records
Consumer group order-projection
stablegeneration 1lag 0RangeAssignor (eager)commit after batchmax.poll.records 10
- projection-1runningprocessed 0merchant-orders-0
- projection-2runningprocessed 0merchant-orders-1
- projection-3runningprocessed 0merchant-orders-2
- projection-4runningprocessed 0merchant-orders-3
| Partition | Owner | Committed | Lag |
|---|---|---|---|
| merchant-orders-0 | projection-1 | none | 0 |
| merchant-orders-1 | projection-2 | none | 0 |
| merchant-orders-2 | projection-3 | none | 0 |
| merchant-orders-3 | projection-4 | none | 0 |
Partition logs
simulated records- merchant-orders-0leader b1ISR 1,2,3HW 0end 0
empty
order-projection: committed none, lag 0
- merchant-orders-1leader b2ISR 2,3,1HW 0end 0
empty
order-projection: committed none, lag 0
- merchant-orders-2leader b3ISR 3,1,2HW 0end 0
empty
order-projection: committed none, lag 0
- merchant-orders-3leader b1ISR 1,2,3HW 0end 0
empty
order-projection: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Acme's own consumer group
stablegeneration 1lag 0RangeAssignor (eager)commit after batchmax.poll.records 10
- acme-projection-1runningprocessed 0merchant-orders-acme-0merchant-orders-acme-1
- acme-projection-2runningprocessed 0merchant-orders-acme-2merchant-orders-acme-3
| Partition | Owner | Committed | Lag |
|---|---|---|---|
| merchant-orders-acme-0 | acme-projection-1 | none | 0 |
| merchant-orders-acme-1 | acme-projection-1 | none | 0 |
| merchant-orders-acme-2 | acme-projection-2 | none | 0 |
| merchant-orders-acme-3 | acme-projection-2 | none | 0 |
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, merchant-orders (4 partitions, RF 3), merchant-orders-acme (4 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.