Skip to content

BuildBreak lab 9 of 15

Broker failure and leader election

Kill brokers under a replicated partition and follow the ISR, the leader epoch and the high watermark through one, two and three failures.

Advanced30 min

Read first: Replication: leaders and followers · In-sync replicas, min.insync.replicas and the high watermark · Partition leadership and leader election · Broker failures and incident response

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
payments: 1 partition, RF 3
Producers
payment-api
Consumer groups
ledger (1)
Guided run
9 stages: build → break → observe → fix → retest

1

Business scenario

The payments topic records every authorised card payment. It runs on three brokers with replication factor 3 and min.insync.replicas=2, and the payment API publishes with acks=all. This week the platform team is replacing the brokers' disks one machine at a time, and a second broker is about to fail on its own in the middle of that. You are on call. Your job is to know, for each combination of dead brokers, whether payments are still accepted, whether any acknowledged payment can be lost, and which of the two settings people will ask you to change at 3 a.m. must not be touched.

2

Starting architecture

Drawing diagram…

The payment API producer sends PaymentAuthorised events with acks=all to the leader replica of partition payments-0 on broker 1. Brokers 2 and 3 hold follower replicas that replicate from the leader. The controller keeps the in-sync replica set and elects the leader. The ledger consumer group reads records below the high watermark from the leader.

flowchart LR
  PA["Payment API producer"] -->|"PaymentAuthorised, acks=all"| L0
  subgraph Cluster["Kafka cluster, 3 brokers, KRaft"]
    L0["payments-0 leader on broker 1"]
    F2["payments-0 follower on broker 2"]
    F3["payments-0 follower on broker 3"]
    CT["Controller"]
    L0 -->|"replicate"| F2
    L0 -->|"replicate"| F3
    CT -->|"keeps the ISR, elects the leader"| L0
  end
  L0 -->|"records below the high watermark"| LG["Ledger consumer group"]
One partition, three replicas. The leader takes the writes, the followers copy them, and the controller decides who is in sync and who leads.

3

Expected behaviour

  • With one broker down, payments are still accepted and read, and no acknowledged payment is lost.
  • With two brokers down, the cluster refuses new payments with an error the producer can see. Nothing that was acknowledged is at risk, and the ledger can still read.
  • A replica that is missing acknowledged records never becomes leader unless somebody explicitly allows it.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 9

Three replicas in sync

Do this

  1. Turn on Payment API publishing (4 payments per second) and press play.
  2. Read the Replicas of payments-0 panel: the leader, the leader epoch, the ISR and each replica's log end offset.

Look for

  • The ISR is [1, 2, 3] and the leader epoch is 0. The followers' log end offsets trail the leader's by a record or two and catch up.
  • The high watermark is the smallest log end offset in the ISR. The producer gets its acknowledgement, and the ledger can read a record, only once the high watermark has passed it.

Not done yet. The simulator checks this for you:produced >= 16 && isr("payments", 0) == 3

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

4 PaymentAuthorised events per second.

Replicas of payments-0

simulated

A record is committed, and visible to consumers, once every replica in the ISR has it. The high watermark marks that point.

Leader
broker 1
Leader epoch
0
In-sync replicas (min 2)
[1, 2, 3]
Acknowledged records on one broker only
0

high watermark 0 · unclean.leader.election.enable=false

brokerstaterolelog end offset
broker 1runningleader0
broker 2runningfollower, in sync0
broker 3runningfollower, in sync0

Simulated metrics

computed by the simulation
Acknowledged to the payment API
0
Acknowledged and lost
0
Leader epoch of payments-0
0
Leader elections
0
Unclean elections
0
Producer retries
0
Refused: not enough in-sync replicas
0
Sends failed for good
0
Brokers up
3
Ledger lag
0records

Partition logs

simulated records
  • payments-0leader b1ISR 1,2,3HW 0end 0

    empty

    ledger: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, payments (1 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.