BuildBreak lab 1 of 15
Producer publishing messages
Publish OrderPlaced events, lose an acknowledged order when a leader dies under acks=1, then make the producer durable and run the same failure again.
Read first: Inside the producer: from send() to the partition log · Acks, retries and the idempotent producer · In-sync replicas, min.insync.replicas and the high watermark
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- orders: 3 partitions, RF 3
- Producers
- order-service
- Consumer groups
- warehouse (1)
- Guided run
- 6 stages: build → break → observe → fix → retest
1
Business scenario
A shop's order service publishes an OrderPlaced event to the orders topic every time a customer checks out. Warehouse, billing and email all start from that event, so an order that never reaches the topic is an order nobody ships. A year ago someone set acks=1 on the producer "for latency" and nobody has looked at it since. Tonight broker 1 is going to die at a bad moment, and support will hear from a customer who has a confirmation page but no parcel.
2
Starting architecture
Drawing diagram…
The order service producer sends OrderPlaced events to the leader replica of partition orders-0 on broker 1. Brokers 2 and 3 hold follower replicas that copy from the leader. The warehouse consumer group reads records below the high watermark.
flowchart LR
OS["Order service producer"] -->|"OrderPlaced"| L0
subgraph Cluster["Kafka cluster with 3 brokers"]
L0["orders-0 leader on broker 1"]
F1["orders-0 follower on broker 2"]
F2["orders-0 follower on broker 3"]
L0 -->|"replicate"| F1
L0 -->|"replicate"| F2
end
L0 -->|"records below the high watermark"| WH["Warehouse consumer group"]3
Expected behaviour
- Every order the order service was told is written (
sendcompleted without an exception) is in theorderslog and stays there. - A broker can stop at any moment: a follower takes over as leader and the warehouse keeps reading, with no order missing and none written twice.
- When the cluster cannot keep a record safely, the producer gets an error it can act on instead of a false success.
4 · Build → break → observe → fix → retest
Guided run
Publish orders
Do this
- Turn on Order service publishing and press play.
- Watch the Partition logs panel: each record first appears on the leader, and the high watermark (HW) follows a moment later, once the followers have copied it.
Look for
- "Acknowledged to the order service" and "Records in the orders log" rise together, and "Acknowledged but lost" stays at 0.
- The producer is running with
acks=1: the simulated log shows it is acknowledged as soon as the leader has the record.
Not done yet. The simulator checks this for you:produced >= 20 && lost == 0
Simulator
Simulation · not a real clusterControls for this stage
5 OrderPlaced events per second, keyed by order id.
Cluster
simulated brokers- order-serviceordersstoppedacks=1idempotence off
- P0 leader
- P1 in sync
- P2 in sync
- P0 in sync
- P1 leader
- P2 in sync
- P0 in sync
- P1 in sync
- P2 leader
- ordersmin.insync.replicas=1
- ordersgroup warehouse1 runninglag 0
Partition logs
simulated records- orders-0leader b1ISR 1,2,3HW 0end 0
empty
warehouse: committed none, lag 0
- orders-1leader b2ISR 2,3,1HW 0end 0
empty
warehouse: committed none, lag 0
- orders-2leader b3ISR 3,1,2HW 0end 0
empty
warehouse: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Simulated metrics
computed by the simulation- Acknowledged to the order service
- 0
- Records in the orders log
- 0
- Acknowledged but lost
- 0
- Duplicates written to the log
- 0
- Duplicates suppressed by the broker
- 0
- Producer retries
- 0
- Failed sends
- 0
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders (3 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.