Skip to content

BuildBreak lab 11 of 15

Hot partition

One large merchant sends most of the traffic under a single key. One partition and one consumer are saturated while the rest idle. Find out why scaling does not help and what each fix costs.

Advanced35 min

Read first: Hot partitions and backpressure · Keys and partitioning: who decides where a record goes · Partition sizing and consumer parallelism · Consumer lag

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
merchant-orders: 4 partitions, RF 3 · merchant-orders-acme: 4 partitions, RF 3
Producers
small-merchants, acme, acme-dedicated
Consumer groups
order-projection (4), order-projection-acme (2)
Guided run
8 stages: build → break → fix → retest

1

Business scenario

A commerce platform publishes every order event of every merchant to the topic merchant-orders, keyed by merchant id so that each merchant's events stay in order. The order projection service (4 instances, one per partition) builds the order pages that merchants see. With a dozen small merchants the load is even. Then Acme, a retailer larger than all the others together, goes live. Acme's order pages fall minutes behind, and so do the pages of two small merchants that have nothing to do with Acme. The dashboard shows the group at 40 percent of its capacity.

2

Starting architecture

Drawing diagram…

Twelve small merchants and one large merchant, Acme, publish to the topic merchant-orders with the merchant id as the record key. The topic has four partitions, each read by one instance of the order projection service, projection-1 to projection-4. All Acme records carry the key merchant-acme.

flowchart LR
  SM["12 small merchants"] -->|"key = merchant id"| T["Topic merchant-orders, 4 partitions"]
  AC["Acme, most of the traffic"] -->|"key = merchant-acme"| T
  T --> P0["partition 0"]
  T --> P1["partition 1"]
  T --> P2["partition 2"]
  T --> P3["partition 3"]
  P0 --> C1["projection-1"]
  P1 --> C2["projection-2"]
  P2 --> C3["projection-3"]
  P3 --> C4["projection-4"]
The key decides the partition. Every Acme record has the same key, so every Acme record goes to the same partition and the same consumer.

3

Expected behaviour

  • Load is spread over the partitions evenly enough that no single consumer is the limit.
  • The events of one order (created, paid, shipped) are processed in that order.
  • A large merchant does not delay the small merchants that happen to share its partition.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 8

A dozen small merchants

Do this

  1. Turn on Small merchants publishing (12 events per second across 12 merchants) and press play.
  2. Look at the per-partition table in the "Lag as time" panel.

Look for

  • Each partition receives a few records per second; each consumer can process 10 per second. Lag is near zero everywhere.
  • "Busiest partition, share of new records" is not far from 25 percent: an even spread over 4 partitions.

Not done yet. The simulator checks this for you:consumed >= 40 && lagAge("order-projection") < 2

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

12 events per second, spread over 12 merchants.

Lag as time (simulated)

simulated

Lag in records says how much is waiting. Lag as time says how stale the newest result is: the age of the oldest record this group has not committed yet.

Lag
0 records
Oldest waiting record
0 s
Arriving
0 /s
Processed
0 /s
Time to drain at these rates
no backlog
Busiest partition, share of new records
0 %
Last backlog
none finished yet

merchant-orders: per partition

partitionownerarriving /slagoldest waiting
merchant-orders-0projection-1000 s
merchant-orders-1projection-2000 s
merchant-orders-2projection-3000 s
merchant-orders-3projection-4000 s

Simulated metrics

computed by the simulation
Records published
0
Records processed
0
Lag, shared topic
0records
Largest partition lag
0records
Oldest waiting record
0s
Busiest partition, share of new records
0%
Processed, shared topic
0/s
Partitions
4
Consumers with no partition
0
Events processed out of order
0
Lag, Acme's own topic
0records

Consumer group order-projection

stablegeneration 1lag 0

RangeAssignor (eager)commit after batchmax.poll.records 10

  • projection-1runningprocessed 0
    merchant-orders-0
  • projection-2runningprocessed 0
    merchant-orders-1
  • projection-3runningprocessed 0
    merchant-orders-2
  • projection-4runningprocessed 0
    merchant-orders-3
Partitions of group order-projection: owner, committed offset and lag
PartitionOwnerCommittedLag
merchant-orders-0 projection-1none0
merchant-orders-1 projection-2none0
merchant-orders-2 projection-3none0
merchant-orders-3 projection-4none0

Partition logs

simulated records
  • merchant-orders-0leader b1ISR 1,2,3HW 0end 0

    empty

    order-projection: committed none, lag 0

  • merchant-orders-1leader b2ISR 2,3,1HW 0end 0

    empty

    order-projection: committed none, lag 0

  • merchant-orders-2leader b3ISR 3,1,2HW 0end 0

    empty

    order-projection: committed none, lag 0

  • merchant-orders-3leader b1ISR 1,2,3HW 0end 0

    empty

    order-projection: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Acme's own consumer group

stablegeneration 1lag 0

RangeAssignor (eager)commit after batchmax.poll.records 10

  • acme-projection-1runningprocessed 0
    merchant-orders-acme-0merchant-orders-acme-1
  • acme-projection-2runningprocessed 0
    merchant-orders-acme-2merchant-orders-acme-3
Partitions of group order-projection-acme: owner, committed offset and lag
PartitionOwnerCommittedLag
merchant-orders-acme-0 acme-projection-1none0
merchant-orders-acme-1 acme-projection-1none0
merchant-orders-acme-2 acme-projection-2none0
merchant-orders-acme-3 acme-projection-2none0

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, merchant-orders (4 partitions, RF 3), merchant-orders-acme (4 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.