Skip to content

BuildBreak lab 1 of 15

Producer publishing messages

Publish OrderPlaced events, lose an acknowledged order when a leader dies under acks=1, then make the producer durable and run the same failure again.

Foundation20 min

Read first: Inside the producer: from send() to the partition log · Acks, retries and the idempotent producer · In-sync replicas, min.insync.replicas and the high watermark

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
orders: 3 partitions, RF 3
Producers
order-service
Consumer groups
warehouse (1)
Guided run
6 stages: build → break → observe → fix → retest

1

Business scenario

A shop's order service publishes an OrderPlaced event to the orders topic every time a customer checks out. Warehouse, billing and email all start from that event, so an order that never reaches the topic is an order nobody ships. A year ago someone set acks=1 on the producer "for latency" and nobody has looked at it since. Tonight broker 1 is going to die at a bad moment, and support will hear from a customer who has a confirmation page but no parcel.

2

Starting architecture

Drawing diagram…

The order service producer sends OrderPlaced events to the leader replica of partition orders-0 on broker 1. Brokers 2 and 3 hold follower replicas that copy from the leader. The warehouse consumer group reads records below the high watermark.

flowchart LR
  OS["Order service producer"] -->|"OrderPlaced"| L0
  subgraph Cluster["Kafka cluster with 3 brokers"]
    L0["orders-0 leader on broker 1"]
    F1["orders-0 follower on broker 2"]
    F2["orders-0 follower on broker 3"]
    L0 -->|"replicate"| F1
    L0 -->|"replicate"| F2
  end
  L0 -->|"records below the high watermark"| WH["Warehouse consumer group"]
One partition of the orders topic: the producer writes to the leader, followers copy from it, consumers read what is replicated.

3

Expected behaviour

  • Every order the order service was told is written (send completed without an exception) is in the orders log and stays there.
  • A broker can stop at any moment: a follower takes over as leader and the warehouse keeps reading, with no order missing and none written twice.
  • When the cluster cannot keep a record safely, the producer gets an error it can act on instead of a false success.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 6

Publish orders

Do this

  1. Turn on Order service publishing and press play.
  2. Watch the Partition logs panel: each record first appears on the leader, and the high watermark (HW) follows a moment later, once the followers have copied it.

Look for

  • "Acknowledged to the order service" and "Records in the orders log" rise together, and "Acknowledged but lost" stays at 0.
  • The producer is running with acks=1: the simulated log shows it is acknowledged as soon as the leader has the record.

Not done yet. The simulator checks this for you:produced >= 20 && lost == 0

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

5 OrderPlaced events per second, keyed by order id.

Cluster

simulated brokers
  • order-serviceordersstoppedacks=1idempotence off
broker 1up
  • P0 leader
  • P1 in sync
  • P2 in sync
broker 2up
  • P0 in sync
  • P1 leader
  • P2 in sync
broker 3up
  • P0 in sync
  • P1 in sync
  • P2 leader
  • ordersmin.insync.replicas=1
  • ordersgroup warehouse1 runninglag 0

Partition logs

simulated records
  • orders-0leader b1ISR 1,2,3HW 0end 0

    empty

    warehouse: committed none, lag 0

  • orders-1leader b2ISR 2,3,1HW 0end 0

    empty

    warehouse: committed none, lag 0

  • orders-2leader b3ISR 3,1,2HW 0end 0

    empty

    warehouse: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Simulated metrics

computed by the simulation
Acknowledged to the order service
0
Records in the orders log
0
Acknowledged but lost
0
Duplicates written to the log
0
Duplicates suppressed by the broker
0
Producer retries
0
Failed sends
0

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders (3 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.