BuildBreak lab 2 of 15
Partition key and ordering
Order status events sent without a key are consumed out of order. Key them by order id, retest, then see what adding a partition does to a key.
Read first: Keys and partitioning: who decides where a record goes · Partitions and offsets
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- order-status: 3 partitions, RF 3
- Producers
- order-service
- Consumer groups
- order-tracking (1)
- Guided run
- 6 stages: build → break → observe → fix → retest
1
Business scenario
The order service publishes a status event each time an order moves on: created, then paid, then shipped. A tracking service consumes the order-status topic and keeps the latest status of every order in its own table, which is what the customer sees on the "Where is my order?" page. The events are published with no key. In testing, with a trickle of traffic, the page was always right. On the first busy evening customers who already have a dispatch email are looking at "paid", and a few at "created".
2
Starting architecture
Drawing diagram…
The order service producer writes created, paid and shipped events to a topic with three partitions. One tracking service consumer reads all three partitions and upserts the latest status per order into the order_tracking table.
flowchart LR
OS["Order service producer"] -->|"created, paid, shipped"| T
subgraph T["Topic order-status"]
P0["Partition 0"]
P1["Partition 1"]
P2["Partition 2"]
end
P0 --> TR["Tracking service consumer"]
P1 --> TR
P2 --> TR
TR -->|"upsert latest status"| DB[("order_tracking table")]3
Expected behaviour
- For any one order, the tracking service processes created before paid before shipped.
- The
order_trackingrow of a shipped order says "shipped" and never goes back to an earlier status. - Events of different orders may be processed in any relative order: nothing depends on that.
4 · Build → break → observe → fix → retest
Guided run
A trickle of traffic looks fine
Do this
- Turn on Order service publishing (2 events per second, no record key) and press play.
- Watch the Partition logs panel: events of the same order land on different partitions.
Look for
- "Out-of-order deliveries" stays at 0 and every row in
order_trackingis correct. - The consumer is faster than the producer, so it reads each event almost as it arrives. Arrival order is hiding the problem.
Not done yet. The simulator checks this for you:consumed >= 12 && outOfOrder == 0
Simulator
Simulation · not a real clusterControls for this stage
2 status events per second.
Partition logs
simulated records- order-status-0leader b1ISR 1,2,3HW 0end 0
empty
order-tracking: committed none, lag 0
- order-status-1leader b2ISR 2,3,1HW 0end 0
empty
order-tracking: committed none, lag 0
- order-status-2leader b3ISR 3,1,2HW 0end 0
empty
order-tracking: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Database
simulated · available| id | order_id | value | written | note |
|---|---|---|---|---|
| no rows yet | ||||
Simulated metrics
computed by the simulation- Events consumed
- 0
- Out-of-order deliveries
- 0
- Orders showing an old status
- 0
- Consumer lag
- 0records
- Keys split across partitions
- 0
- Partitions
- 3
Consumer group order-tracking
stablegeneration 1lag 0RangeAssignor (eager)commit after batchmax.poll.records 10
- tracking-1runningprocessed 0order-status-0order-status-1order-status-2
| Partition | Owner | Committed | Lag |
|---|---|---|---|
| order-status-0 | tracking-1 | none | 0 |
| order-status-1 | tracking-1 | none | 0 |
| order-status-2 | tracking-1 | none | 0 |
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, order-status (3 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.