BuildBreak lab 9 of 15
Broker failure and leader election
Kill brokers under a replicated partition and follow the ISR, the leader epoch and the high watermark through one, two and three failures.
Read first: Replication: leaders and followers · In-sync replicas, min.insync.replicas and the high watermark · Partition leadership and leader election · Broker failures and incident response
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- payments: 1 partition, RF 3
- Producers
- payment-api
- Consumer groups
- ledger (1)
- Guided run
- 9 stages: build → break → observe → fix → retest
1
Business scenario
The payments topic records every authorised card payment. It runs on three brokers with replication factor 3 and min.insync.replicas=2, and the payment API publishes with acks=all. This week the platform team is replacing the brokers' disks one machine at a time, and a second broker is about to fail on its own in the middle of that. You are on call. Your job is to know, for each combination of dead brokers, whether payments are still accepted, whether any acknowledged payment can be lost, and which of the two settings people will ask you to change at 3 a.m. must not be touched.
2
Starting architecture
Drawing diagram…
The payment API producer sends PaymentAuthorised events with acks=all to the leader replica of partition payments-0 on broker 1. Brokers 2 and 3 hold follower replicas that replicate from the leader. The controller keeps the in-sync replica set and elects the leader. The ledger consumer group reads records below the high watermark from the leader.
flowchart LR
PA["Payment API producer"] -->|"PaymentAuthorised, acks=all"| L0
subgraph Cluster["Kafka cluster, 3 brokers, KRaft"]
L0["payments-0 leader on broker 1"]
F2["payments-0 follower on broker 2"]
F3["payments-0 follower on broker 3"]
CT["Controller"]
L0 -->|"replicate"| F2
L0 -->|"replicate"| F3
CT -->|"keeps the ISR, elects the leader"| L0
end
L0 -->|"records below the high watermark"| LG["Ledger consumer group"]3
Expected behaviour
- With one broker down, payments are still accepted and read, and no acknowledged payment is lost.
- With two brokers down, the cluster refuses new payments with an error the producer can see. Nothing that was acknowledged is at risk, and the ledger can still read.
- A replica that is missing acknowledged records never becomes leader unless somebody explicitly allows it.
4 · Build → break → observe → fix → retest
Guided run
Three replicas in sync
Do this
- Turn on Payment API publishing (4 payments per second) and press play.
- Read the Replicas of payments-0 panel: the leader, the leader epoch, the ISR and each replica's log end offset.
Look for
- The ISR is [1, 2, 3] and the leader epoch is 0. The followers' log end offsets trail the leader's by a record or two and catch up.
- The high watermark is the smallest log end offset in the ISR. The producer gets its acknowledgement, and the ledger can read a record, only once the high watermark has passed it.
Not done yet. The simulator checks this for you:produced >= 16 && isr("payments", 0) == 3
Simulator
Simulation · not a real clusterControls for this stage
4 PaymentAuthorised events per second.
Replicas of payments-0
simulatedA record is committed, and visible to consumers, once every replica in the ISR has it. The high watermark marks that point.
- Leader
- broker 1
- Leader epoch
- 0
- In-sync replicas (min 2)
- [1, 2, 3]
- Acknowledged records on one broker only
- 0
high watermark 0 · unclean.leader.election.enable=false
| broker | state | role | log end offset |
|---|---|---|---|
| broker 1 | running | leader | 0 |
| broker 2 | running | follower, in sync | 0 |
| broker 3 | running | follower, in sync | 0 |
Simulated metrics
computed by the simulation- Acknowledged to the payment API
- 0
- Acknowledged and lost
- 0
- Leader epoch of payments-0
- 0
- Leader elections
- 0
- Unclean elections
- 0
- Producer retries
- 0
- Refused: not enough in-sync replicas
- 0
- Sends failed for good
- 0
- Brokers up
- 3
- Ledger lag
- 0records
Partition logs
simulated records- payments-0leader b1ISR 1,2,3HW 0end 0
empty
ledger: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, payments (1 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.