Skip to content

Path 2 · lesson 3 of 8

Intermediate17 minApache Kafka 4.4 (KRaft only) · Java 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 (kafka-clients 3.9)

Replication: leaders and followers

How each partition is copied to several brokers, what the leader and followers each do, what the leader epoch is for, and what a client sees when leadership moves.

What you will learn

  • Explain what the replication factor buys and what it costs
  • Describe how a follower copies the leader's log and how the leader knows how far each follower has got
  • Say what a leader epoch is and the two problems it solves
  • Predict what a producer and a consumer experience when a partition leader fails or is shut down cleanly
  • Spread replicas across racks or zones, and let consumers read from a replica in their own zone

Before you start

The concept

Replication keeps more than one copy of every partition, each on a different broker. The replication factor is the number of copies: with a replication factor of 3, payments-0 exists on three brokers. The copies are called replicas, and the unit of replication is the partition, not the topic and not the broker.

At any moment one replica is the leader and the rest are followers. Every write for the partition goes to the leader, which decides the order of records by appending them to its log. Followers do one thing: copy the leader's log, in the same order, to their own disk. The result is that all replicas hold the same records at the same offsets, with a follower possibly missing the last few that it has not fetched yet.

If the leader's broker fails, the controller picks one of the followers that is fully caught up and makes it the new leader. Clients find the new leader and carry on. With a replication factor of N, a partition can lose N minus 1 brokers and still have a copy of every record that was fully replicated. Which followers count as caught up, and what fully replicated means for a producer, is the subject of the next lesson.

Leaders and followers

Simulation · not a real cluster

Click a partition, then each of its replicas, to see which one leads and how far each follower has copied. Compare a topic with three replicas to one with two.

Why it exists

Without replication, a partition is one directory on one disk. When that broker restarts for an upgrade the partition is unavailable, and when the disk dies the data is gone. A system that other services depend on for every order and payment cannot have planned maintenance turn into downtime, so Kafka treats replication as the normal case: an unreplicated topic is simply a replicated topic with a factor of one.

Why one leader per partition, and not writes to any replica? A log is useful because everyone agrees on the order of its records. The cheapest way to get agreement is to let a single node choose the order and have the others copy it. As long as the leader is alive there is nothing to vote on.

Why do followers pull, instead of the leader pushing? A follower fetches from the leader with the same kind of request a consumer uses. A follower that has fallen behind asks for a large batch and catches up quickly; one that is up to date waits on the leader for new data. The leader does not need to track delivery to each follower, and it learns each follower's progress for free from the offset in its next request.

Why not majority voting, as Raft-style systems do? With a majority quorum, surviving one failure takes three copies and surviving two takes five. Kafka's design keeps a dynamic set of in-sync replicas instead, and with that model N replicas can survive N minus 1 failures without losing committed records. For large volumes of data that is a large saving in disk and network. Kafka does use a majority quorum where the data is small and correctness is everything: the controller's metadata log.

How it works inside

How a follower copies the log

  1. The follower sends the leader a fetch request containing the offset it wants next, which is its own log end offset.
  2. The leader answers with the records from that offset onwards and with its current high watermark. If there is nothing new, the leader holds the request for up to replica.fetch.wait.max.ms (500 ms) before answering empty.
  3. The follower appends the records to its log, in the same order and with the same offsets, and records the high watermark it was told.
  4. The follower's next request carries a higher offset. That is the leader's evidence that the follower has stored everything below it, and what lets the leader advance the high watermark.

A broker runs fetcher threads for this (num.replica.fetchers per source broker, 1 by default), and one fetch request covers many partitions led by the same broker. Replication is therefore continuous and batched, not one round trip per record.

Leader epoch

Every time a partition gets a new leader, the controller increases the partition's leader epoch, a number that identifies one continuous period of leadership. The leader stamps the epoch on every record batch it appends, and each replica keeps a small list of which offset each epoch started at. The epoch does two jobs.

JobThe problemHow the epoch solves it
Fencing a stale leaderA broker that was cut off may still believe it leads the partition and keep accepting requests.Requests carry the epoch the sender believes is current. A broker or client holding an older epoch is rejected with FENCED_LEADER_EPOCH and has to refresh its metadata.
Truncating a diverged logA returning replica may hold records at the end of its log that the new leader never had.Before fetching, the replica asks the leader where its last epoch ended. It cuts its log back to that offset and then copies from the leader, so every replica converges on the leader's history.

Consumers use the same information: a consumer remembers the epoch of the last record it read and can detect that the log was cut back underneath its position instead of silently reading different records at the same offsets. Leader election itself is covered in partition leadership and election.

What a client sees when the leader moves

Clean shutdown or planned move

  • The broker asks the controller to move its leaderships away before it stops (controlled.shutdown.enable, on by default)
  • Each partition is without a leader for a few milliseconds
  • Producers get one NOT_LEADER_OR_FOLLOWER error per affected partition, refresh metadata and resend
  • The application sees a slightly slower send and nothing else

Crash or network loss

  • The controller notices only when the broker's heartbeats have been missing for broker.session.timeout.ms (9 seconds)
  • Until then, requests to the dead leader time out or fail to connect
  • Producers buffer and retry until a new leader is elected, then resend
  • The application sees sends that take seconds, and failures only if delivery.timeout.ms (2 minutes) runs out

For a consumer, a leader change is an interrupted fetch followed by a metadata refresh and a fetch to the new leader from the same position. It is not a consumer group rebalance: partition assignments within the group do not change because a broker failed. The one case where the group itself pauses is when the failed broker was the group's coordinator, and the group has to find the new one.

Racks, zones and reading from followers

Three replicas on three brokers in the same rack or availability zone share one failure. Setting broker.rack on each broker tells the controller where it is, and replica assignment then spreads each partition's replicas over as many racks as possible: a partition spans the smaller of the number of racks and the replication factor. Use the same number of brokers in each rack, or the racks with fewer brokers end up holding more replicas each.

By default consumers fetch from the leader, which in a multi-zone cluster is often in another zone, and cross-zone traffic is usually billed. Follower fetching (KIP-392) lets a consumer fetch from a follower instead. It needs two settings: replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector on the brokers, and client.rack on the consumer set to the same value its local brokers use for broker.rack. The leader then points the consumer at a caught-up replica in its own rack. The consumer still reads only up to the high watermark, and a follower learns the high watermark slightly after the leader, so expect a small increase in end-to-end latency. Producers always write to the leader.

Architecture

Drawing diagram…

A producer writes to broker 1 in zone A, the leader of payments partition 0 at epoch 7. Brokers 2 and 3 in zones B and C fetch from the leader. A consumer in zone A fetches from the leader and a consumer in zone B fetches from the follower in its own zone. The controller chooses the leader and distributes metadata.

flowchart TB
  P["Producer"] -->|"produce"| L
  subgraph ZA["Zone A"]
    L["Broker 1, payments-0 LEADER, epoch 7"]
    CA["Consumer in zone A"]
  end
  subgraph ZB["Zone B"]
    F2["Broker 2, payments-0 follower"]
    CB["Consumer in zone B, client.rack is zone B"]
  end
  subgraph ZC["Zone C"]
    F3["Broker 3, payments-0 follower"]
  end
  F2 -->|"fetch from its log end"| L
  F3 -->|"fetch from its log end"| L
  CA -->|"fetch"| L
  CB -->|"fetch from nearest replica"| F2
  K["Active controller"] -. "chooses leader and epoch" .-> L
  K -. "metadata" .-> F2
  K -. "metadata" .-> F3
One partition, three replicas in three zones: writes go to the leader, followers pull, consumers can read locally.
  • All arrows for writes end at the leader. The followers' arrows point towards the leader because followers initiate replication.
  • Each replica is in a different zone, so losing a whole zone leaves two copies of the partition.
  • The consumer in zone B reads from broker 2 because follower fetching is configured. Without it, that consumer would fetch from broker 1 across zones.
  • The controller is not on the data path. It appears only to say who the leader is and which epoch is current.

Message flow

Drawing diagram…

The producer sends a batch to broker 1, which crashes. After nine seconds without heartbeats the controller makes broker 2 the leader at epoch 8. Broker 3 aligns its log with broker 2. The producer refreshes metadata and resends the batch to broker 2. When broker 1 returns it truncates its log to where epoch 7 ended and follows broker 2.

sequenceDiagram
  participant P as Producer
  participant B1 as Broker 1 old leader
  participant K as Controller
  participant B2 as Broker 2
  participant B3 as Broker 3
  P->>B1: Produce batch 57 to payments-0 at epoch 7
  Note over B1: Broker 1 crashes before answering
  Note over K: No heartbeat from broker 1 for 9 seconds
  K->>B2: You lead payments-0 at epoch 8
  K->>B3: Follow broker 2 at epoch 8
  B3->>B2: Where did epoch 7 end
  B2-->>B3: At offset 4120
  B3->>B2: Fetch from offset 4120
  P->>B3: Metadata request after timeout
  B3-->>P: Leader of payments-0 is broker 2
  P->>B2: Produce batch 57 again
  B2-->>P: Acknowledged, no duplicate
  Note over B1: Broker 1 restarts as a follower
  B1->>B2: Where did epoch 7 end
  B2-->>B1: At offset 4120
  Note over B1: Truncates to 4120 and fetches from broker 2
A leader crash: detection, a new leader at a higher epoch, a producer retry, and the old leader rejoining as a follower.
  1. The gap between the crash and the election is mostly detection time. During it the producer holds the batch in memory and retries.
  2. The producer resends the same batch with the same sequence number. Whether or not broker 1 wrote it before crashing, the log ends up with exactly one copy.
  3. Broker 3 and later broker 1 both ask the new leader where epoch 7 ended before fetching. Anything they hold beyond that offset was never part of the agreed log and is cut off.
  4. Broker 1 comes back as a follower. It may be given leadership again later by a preferred leader election, once it has caught up.

In a real system

A typical payments platform runs six brokers across three availability zones, two per zone, with broker.rack set to the zone name. The payments topic has 12 partitions and a replication factor of 3, so every partition has exactly one replica in each zone.

EventWhat happens to paymentsWhat the payment service sees
Rolling restart of one brokerIts leaderships move away before it stops; it catches up after restartA handful of retried batches, no errors
One broker crashesAbout two partitions lose their leader for roughly ten secondsSends to those partitions are delayed, then succeed
A whole zone is lostA third of the partitions elect a new leader; all partitions run on two replicasA delay on some partitions; no data loss for fully replicated records
Two zones are lostEvery partition is down to one replicaDepends on min.insync.replicas, covered in the next lesson

The fraud-scoring consumers run in all three zones and set client.rack to their zone, so each reads from the replica next to it and the cluster's cross-zone traffic is mostly replication. The producers cannot do the same: a write always goes to the leader, wherever it is.

The cost is visible on the bill and in the latency: three times the disk, every record crossing zones twice for replication, and, with acks=all, each send waiting for the followers. That is why a clickstream topic in the same cluster might run with a replication factor of 2, while payments never would.

Java and Spring Boot

Replication is configured on the topic and the brokers, not in application code. What the application controls is how it creates the topic, how long its producer keeps retrying through a failover, and whether its consumers read from a nearby replica.

src/main/java/com/example/payments/kafka/PaymentTopicConfig.javaJava
package com.example.payments.kafka;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.common.config.TopicConfig;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;@Configurationpublic class PaymentTopicConfig {    public static final String PAYMENTS = "payments";    @Bean    public NewTopic paymentsTopic() {        return TopicBuilder.name(PAYMENTS)                .partitions(12)                .replicas(3)                .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2")                .build();    }}

Why it is written this way

  • replicas(3) is the replication factor. Topic creation fails if the cluster has fewer than three brokers available, which is the right outcome: silently creating fewer copies would be worse.
  • The application states how many replicas it wants and leaves placement to the controller, which spreads them across racks when broker.rack is set.
  • min.insync.replicas=2 is set here because a replication factor without it does not give a durability guarantee. The next lesson explains why.
  • KafkaAdmin does not change the replication factor of a topic that already exists. That needs a partition reassignment.

Configuration

Replication is mostly controlled by a few topic and broker settings, plus two client settings that decide how a failover feels to the application. Defaults are from the Kafka 4.4 configuration reference.

default.replication.factorDefault 1
What it does
Broker setting. Replication factor for topics created without one, including auto-created topics.
When to change it
Set it to 3 on any production cluster. A factor of 3 lets one broker be down for maintenance while still tolerating one unplanned failure.
broker.rackDefault null
What it does
Broker setting. The rack, zone or other failure domain of this broker, used to spread replicas and to select a nearby replica for consumers.
When to change it
Set it on every broker in a multi-rack or multi-zone deployment, with the same number of brokers in each.
replica.selector.classDefault null (consumers fetch from the leader)
What it does
Broker setting. The class that picks which replica a consumer should fetch from.
When to change it
Set it to org.apache.kafka.common.replica.RackAwareReplicaSelector to let consumers with client.rack read from a replica in their own rack.
client.rackDefault empty
What it does
Consumer setting. The rack this client runs in. It corresponds to the broker setting broker.rack.
When to change it
Set it per zone when cross-zone fetch traffic matters. It does nothing without the broker-side selector.
replica.fetch.wait.max.msDefault 500
What it does
Broker setting. How long the leader may hold a follower's fetch request when there is no new data.
When to change it
Keep it well below replica.lag.time.max.ms, or followers of quiet partitions will look as if they have stopped fetching.
num.replica.fetchersDefault 1
What it does
Broker setting. Fetcher threads per source broker used to replicate from it.
When to change it
Increase it if followers cannot keep up on a busy cluster and the leader's disk and network are not the limit. It costs CPU and memory on both sides.
broker.session.timeout.msDefault 9000 (9 seconds)
What it does
Broker and controller setting. How long the controller waits without a heartbeat before treating a broker as gone. Heartbeats are sent every broker.heartbeat.interval.ms (2 seconds).
When to change it
This is most of the unavailability after a crash. Lowering it speeds up failover and makes false alarms during long pauses more likely.
delivery.timeout.msDefault 120000 (2 minutes)
What it does
Producer setting. Upper bound on the time between send() returning and the record being reported as stored or failed, including all retries.
When to change it
It must be longer than a leader election takes in your cluster. Shorten it only if the caller has a better plan than waiting.
controlled.shutdown.enableDefault true
What it does
Broker setting. On a clean stop, move this broker's leaderships to other replicas before shutting down.
When to change it
Leave it on, and stop brokers with a normal termination signal so that it can run. A killed process skips it.
auto.leader.rebalance.enableDefault true
What it does
Broker setting. Periodically (every leader.imbalance.check.interval.seconds, 300) move leadership back to each partition's preferred replica when leaders have become unevenly spread.
When to change it
Leave it on, so that a broker that was restarted takes its share of leaderships back without manual work.

Hands-on exercise

Kill a leader and follow the failover

Watch one partition go through a leader failure and recovery, and separate what changes (leader, epoch, the in-sync list) from what does not (offsets, the other partitions).

Steps · 0 of 7 done

Three replicas, one failure

Simulation · not a real cluster

Stop the broker that leads a partition and watch a follower take over. Click replicas to compare their log end offsets and the leader epoch before and after.

Take it further in the lab: Broker failure and leader election

What can go wrong

Replication factor 1 on a topic that matters
Symptom
During a routine broker restart some partitions show Leader: none, producers time out, and consumers of those partitions stall until the broker is back.
Cause
The topic was created without a replication factor on a cluster where default.replication.factor is still 1, or it was auto-created. Each partition has a single copy.
Fix
Set the replication factor explicitly, raise the broker default to 3, and increase existing topics with a partition reassignment. A single copy also means a lost disk is lost data.
All replicas in one failure domain
Symptom
One availability zone or rack goes down and some partitions lose every replica, although the replication factor is 3.
Cause
broker.rack is not set, so replicas were assigned without regard to location and some partitions ended up entirely in one zone.
Fix
Set broker.rack on every broker and reassign existing partitions so that their replicas span zones. New topics are then spread automatically.
Acknowledged writes lost with acks=1
Symptom
After a broker crash, a few records that the producer logged as sent are missing from the topic. No errors were reported.
Cause
With acks=1 the leader acknowledges as soon as it has written the record locally. If it fails before a follower has fetched it, the new leader never had it.
Fix
Use acks=all (the default) with min.insync.replicas of at least 2 for data that must not be lost.
Sends fail during an ordinary failover
Symptom
Every broker restart produces a burst of TimeoutException in the application, although the cluster recovers within seconds.
Cause
delivery.timeout.ms was lowered to a few seconds, or retries was set to 0, so the producer gives up before the new leader is elected.
Fix
Leave retries unset and keep delivery.timeout.ms longer than the worst-case failover. If the caller cannot wait that long, make the send asynchronous instead of shortening the timeout.
Follower fetching configured on one side only
Symptom
client.rack is set on consumers but cross-zone traffic has not dropped.
Cause
The brokers still use the default replica selector, which always returns the leader, or client.rack does not match any broker.rack value exactly.
Fix
Set replica.selector.class to the rack-aware selector on all brokers and make the rack strings identical on both sides.

How to troubleshoot

Need to see who leads a partition and where its replicas are

Check

kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --topic payments
How to read it
Per partition: Leader is the broker id taking writes, Replicas is the assigned list in preference order (the first one is the preferred leader), and Isr lists the replicas currently in sync. A leader that is not the first replica means leadership has moved since assignment.
Some partitions have fewer healthy copies than they should

Check

kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --under-replicated-partitions
How to read it
Lists partitions whose Isr is smaller than Replicas. The broker id missing from every line is the one that is down or lagging. The matching metric is kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions, which should be 0.
One broker is much busier than the others

Check

kafka.server:type=ReplicaManager,name=LeaderCount on each broker
How to read it
Leader counts should be roughly even. A broker with far more leaders took them over during a failure and has not handed them back. A preferred leader election (kafka-leader-election.sh --election-type preferred --all-topic-partitions) restores the balance.
Producer logs show warnings about NOT_LEADER_OR_FOLLOWER

Check

kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs, and the broker logs around the same time
How to read it
A few warnings followed by successful sends are the normal trace of a leader change. A steady stream means leaders keep moving: look for a broker that is repeatedly losing its session with the controller.
A follower never catches up

Check

kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica on the follower broker
How to read it
The largest number of records this broker is behind on any partition it follows. It should fall steadily after a restart. A value that stays high or grows means the follower cannot fetch as fast as the leader is written to.

Key takeaways

  • Each partition has one leader and replication factor minus one followers, on different brokers. Writes always go to the leader.
  • Followers pull from the leader with fetch requests, and the offset in each request tells the leader how far that follower has got.
  • The leader epoch increases with every leader change. It fences stale leaders and tells a returning replica where to cut its log.
  • A leader change is handled inside the clients: an error, a metadata refresh, a retry. With idempotence on, the retry adds no duplicates.
  • A crash costs roughly the broker session timeout before a new leader exists. A clean shutdown costs milliseconds.
  • Set broker.rack so that replicas span failure domains, and use follower fetching to keep consumer reads local.
  • Replication alone does not make an acknowledged write safe. That depends on acks and the in-sync replicas.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

payments has replication factor 3. Partition 4 has replicas on brokers 1, 2 and 3, all in sync, with broker 1 leading. The producer uses acks=all, idempotence and the default delivery.timeout.ms. Broker 1 loses power. For about 11 seconds the producer's metrics show retries for partition 4, then sends succeed again. No exceptions reached the application.

What should the on-call engineer conclude about the records sent during those 11 seconds?

Scenario 2 of 3

A producer writes to audit-0 with acks=1. The leader, broker 2, has acknowledged records up to offset 520. Its followers have fetched up to offset 510 when broker 2's disk fails permanently. A follower becomes leader, and the next record the producer sends is acknowledged at offset 511.

What happened to the ten records the old leader acknowledged at offsets 511 to 520?

Scenario 3 of 3

A cluster has two brokers in each of three zones, with broker.rack set. Consumers of clickstream run in all three zones and the cloud bill shows heavy cross-zone traffic from brokers to consumers. A colleague has set client.rack on every consumer to its zone name, and nothing changed.

What is the missing step?

References

References: Apache Kafka documentation: Design, Replication ↗ · Apache Kafka documentation: Basic Kafka operations (graceful shutdown, balancing leadership, racks) ↗ · Apache Kafka documentation: Broker configs ↗ · Apache Kafka documentation: Message format (partition leader epoch in the record batch) ↗ · KIP-101: Use leader epoch rather than high watermark for truncation ↗ · KIP-392: Allow consumers to fetch from closest replica ↗ · KIP-320: Allow fetchers to detect and handle log truncation ↗ · Spring for Apache Kafka reference: Sending messages ↗ · Report a problem