Skip to content

Path 2 · lesson 4 of 8

Intermediate18 minApache Kafka 4.4 (KRaft only) · Java 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 (kafka-clients 3.9)

In-sync replicas, min.insync.replicas and the high watermark

Exactly what in sync means, how the ISR shrinks and grows, how the high watermark follows it, and what acks=all with min.insync.replicas really guarantees.

What you will learn

  • State the two conditions a replica must meet to be in sync, and why lag is measured in time
  • Explain how the in-sync set shrinks and grows and how the high watermark depends on it
  • Work out from a partition's replica state whether a producer's write will succeed, wait or fail
  • Work out from the same state what a consumer can and cannot read
  • Choose replication factor, min.insync.replicas and acks for a topic and say what failure each choice survives
  • Explain what unclean leader election gives up and when a team might still enable it

Before you start

The concept

A partition's in-sync replicas (the ISR) are the replicas that currently hold a complete copy of the log. The leader is always a member. A follower is in sync when two things are true:

  1. Its broker has a live session with the controller, kept up by heartbeats. A broker whose heartbeats stop for broker.session.timeout.ms is removed from every ISR it was in.
  2. It has caught up to the leader's log end offset at some point within the last replica.lag.time.max.ms (30 seconds by default). A follower that has stopped fetching, or is fetching but never reaches the end, is removed by the leader after that time.

The second condition is about time, not about a number of records. A follower that is 50,000 records behind during a burst but reaches the end of the log every few seconds stays in sync. A follower that is only 10 records behind but has not reached the end for 30 seconds does not. There is no setting for how many records behind a follower may be.

The ISR matters because of what Kafka builds on it. A record is committed once every replica in the ISR has it. The high watermark is the offset below which every record is committed, and consumers are only ever given records below it. And when a leader fails, the controller picks the new leader from the ISR, so the new leader is guaranteed to have every committed record. Kafka's promise is exactly this: a committed record is not lost as long as at least one in-sync replica survives.

A partition and its in-sync set

Simulation · not a real cluster

Click a partition to see its replicas, which of them are in sync, and where the high watermark is compared with each replica's log end offset.

Why it exists

A replicated log has to answer one question for every write: how many copies must exist before the producer is told it succeeded? Each fixed answer has a problem.

RuleWhat goes wrong
Acknowledge after the leader has writtenThe record exists on one disk. If the leader dies before a follower fetches it, an acknowledged record is gone.
Wait for every assigned replicaOne slow or dead follower blocks all writes to the partition until someone intervenes.
Wait for a fixed majoritySafe and fast, but surviving one failure needs three copies and surviving two needs five. Expensive for bulk data.

Kafka's answer is to wait for every replica that is currently keeping up, and to stop waiting for one that is not. The set is dynamic: a follower that falls behind is dropped from it and writes continue with the rest; when it has caught up it is added back. That gives the safety of waiting for all copies without letting one bad broker stop the partition.

The dynamic set has an obvious weakness. It can shrink all the way down to the leader alone, and at that point waiting for all in-sync replicas means waiting for nobody. min.insync.replicas exists to put a floor under it: below that size, the partition refuses writes that asked for full acknowledgement instead of quietly accepting them onto a single disk.

How it works inside

How the ISR shrinks

There are two routes. If a follower's broker dies or loses contact, the controller notices the missing heartbeats and removes that broker from the ISR of every partition it hosts. If the broker is alive but a follower is slow, the leader notices: it tracks when each follower last reached its log end offset, and when that was more than replica.lag.time.max.ms ago it asks the controller to remove the follower. Either way the change is written to the cluster metadata by the controller, so there is one authoritative ISR per partition.

Between a follower stalling and its removal, the leader is still waiting for it. Writes with acks=all to that partition are not acknowledged and the high watermark does not move, for up to 30 seconds with the default setting. A stuck follower therefore shows up as a pause in produce acknowledgements and in consumer delivery, followed by an ISR shrink, after which both resume.

How the ISR grows

A follower outside the ISR keeps fetching. When it returns after a failure it first aligns its log with the leader using the leader epoch (see replication), then copies what it is missing. Only when it has fully caught up to the leader's log end does the leader ask the controller to add it back. A replica is never re-admitted on the strength of having been in sync before: it has to re-sync completely, even if it lost unflushed data in its crash.

The high watermark

The leader knows each follower's log end offset from that follower's fetch requests. The high watermark is the lowest log end offset among the in-sync replicas: everything below it is on all of them. The leader includes the current value in its fetch responses, so followers learn it one round trip later. Since Kafka 4.0 there is a second condition: the high watermark only advances while the ISR has at least min.insync.replicas members. Earlier versions let it advance with a smaller ISR.

acks=all meets min.insync.replicas

acks=all on the producer means: do not acknowledge until every replica in the ISR at this moment has the record. min.insync.replicas on the topic means: if the ISR has fewer members than this, reject acks=all writes. For a topic with replication factor 3 and min.insync.replicas=2:

ISR sizeProducer with acks=allProducer with acks=1Consumers
3Acknowledged when all three replicas have the recordAcknowledged when the leader has written itRead up to the high watermark, which keeps advancing
2Acknowledged when both replicas have it. No margin left for another failureAs aboveAs above
1 (leader only)Rejected with NotEnoughReplicasException. The producer retries until delivery.timeout.ms, then the send failsStill appended and acknowledged by the leaderCan read everything already below the high watermark. In Kafka 4.x the high watermark stops, so new acks=1 records are not delivered until the ISR recovers
0 (no leader)Fails: there is nobody to send toFailsCannot fetch from this partition at all
Replication factor 3, min.insync.replicas 2.

A related error, NotEnoughReplicasAfterAppendException, means the leader had already written the batch when the ISR dropped below the minimum. The record is in the leader's log but was not acknowledged. The producer retries it, and with idempotence enabled the retry does not create a second copy.

Durability against availability

Favour durability

  • Replication factor 3, min.insync.replicas=2, producers use acks=all
  • unclean.leader.election.enable=false
  • An acknowledged record is on at least two brokers
  • With two brokers down, producers get errors and must hold or reject work
  • Suits payments, orders, ledgers, anything used as a source of truth

Favour availability

  • min.insync.replicas=1, often acks=1
  • Sometimes unclean.leader.election.enable=true
  • Writes continue while any one replica is alive
  • Records acknowledged by a lone leader are lost if that broker fails
  • Suits metrics, clickstream, logs, where a gap is cheaper than back-pressure

Setting min.insync.replicas equal to the replication factor is rarely what anyone wants: every broker restart then takes one replica out of the ISR and stops writes to its partitions. The minimum should leave room for at least one replica to be absent.

When every in-sync replica is gone

If all members of the ISR are down, the partition has no leader. The controller can wait for one of them to return, which keeps the partition offline for as long as that takes, or it can promote a replica that was not in sync, which brings the partition back immediately with whatever that replica has. The second choice is an unclean leader election. The out-of-sync replica's log becomes the truth: committed records it never received are gone, the other replicas truncate to match it when they return, and consumers that had read further may find their offsets pointing past the end of the log. It is off by default (unclean.leader.election.enable=false), and has been since 0.11.

Kafka 4.x narrows the gap between those two choices with eligible leader replicas (KIP-966), enabled by default on clusters created with 4.1 or later. Because the high watermark no longer advances while the ISR is below the minimum, a replica that leaves the ISR at that point still holds everything up to the high watermark. The controller remembers such replicas and can elect one of them safely when the ISR is empty. Election order and preferred leaders are covered in partition leadership and election.

ISR / acks

Simulation, not a real cluster

Animated walk-through of a write under acks=all as the in-sync set shrinks.

Step 1 · The pattern

A leader and its followers

Every partition has a leader that takes writes and followers that copy them. If the leader dies, a follower that is caught up takes over.

  • message
  • being copied to follower
  • confirmed
Open the full animated lesson

Architecture

Drawing diagram…

A state diagram for a follower replica. It starts catching up, becomes in sync when it reaches the leader's log end, becomes out of sync if it has not caught up within replica.lag.time.max.ms, goes offline if its broker's heartbeats are lost, and returns through catching up after a restart.

stateDiagram-v2
  state "In sync" as InSync
  state "Out of sync but fetching" as Lagging
  state "Offline" as Offline
  state "Catching up" as CatchingUp
  [*] --> CatchingUp: replica assigned or broker started
  CatchingUp --> InSync: reached the leader log end, leader asks controller to expand ISR
  InSync --> Lagging: not caught up for replica.lag.time.max.ms, leader asks controller to shrink ISR
  InSync --> Offline: broker heartbeats lost, controller removes it from ISR
  Lagging --> InSync: caught up to the leader log end again
  Lagging --> Offline: broker stops
  Offline --> CatchingUp: broker restarts, truncates by leader epoch, fetches
  note right of InSync
    Counts towards acks=all
    Holds back the high watermark
    Eligible to become leader
  end note
  note right of Lagging
    Not waited for
    Not eligible in a clean election
  end note
The life of one follower replica: in and out of the in-sync set.
  • Membership changes in both directions go through the controller, which records the new ISR in the cluster metadata. The leader proposes; the controller decides.
  • Only replicas in the In sync state are waited for by acks=all, and only they limit the high watermark. That is why removing a slow replica lets writes flow again.
  • The path back in always passes through Catching up. There is no shortcut from Offline to In sync.
  • A partition is under-replicated whenever at least one of its replicas is in a state other than In sync.

Message flow

Drawing diagram…

With three in-sync replicas, the leader acknowledges record A after both followers have fetched past it. Then broker 3 stalls. Record B is appended and fetched by broker 2, but the leader waits for broker 3 until thirty seconds have passed, asks the controller to shrink the ISR to brokers 1 and 2, raises the high watermark and acknowledges.

sequenceDiagram
  participant P as Producer acks=all
  participant B1 as Broker 1 leader
  participant B2 as Broker 2 follower
  participant B3 as Broker 3 follower
  participant K as Controller
  Note over B1,B3: ISR is 1 2 3 and min.insync.replicas is 2
  P->>B1: Produce record A
  Note over B1: Appended at offset 120
  B2->>B1: Fetch from 120
  B3->>B1: Fetch from 120
  B2->>B1: Fetch from 121
  B3->>B1: Fetch from 121
  Note over B1: All of the ISR has offset 120, high watermark is 121
  B1-->>P: Acknowledged offset 120
  Note over B3: Broker 3 disk stalls and it stops fetching
  P->>B1: Produce record B
  Note over B1: Appended at offset 121
  B2->>B1: Fetch from 122
  Note over B1: Waiting for broker 3, no acknowledgement yet
  Note over B1: 30 seconds pass without broker 3 catching up
  B1->>K: Shrink ISR of payments-0 to 1 2
  K-->>B1: ISR is now 1 2
  Note over B1: High watermark is 122
  B1-->>P: Acknowledged offset 121
The same write twice: once with a healthy ISR, once while a follower stalls and is removed.
  1. The leader learns that a follower has offset 120 from that follower's next fetch, which asks for 121. Acknowledgement of record A follows the slower of the two followers.
  2. While broker 3 is stalled but still in the ISR, record B is on two brokers and still not acknowledged. Being in the ISR means being waited for.
  3. After the shrink, the ISR has two members, which meets min.insync.replicas=2, so the write is acknowledged and consumers can now read offset 121.
  4. If broker 2 then failed as well, the ISR would be the leader alone, below the minimum, and the next acks=all write would be rejected instead of being accepted onto one disk.
  5. The producer's request.timeout.ms is also 30 seconds by default, so during such a stall the producer may time out and resend the batch. With idempotence on, the resend is recognised and not written twice.

In a real system

A typical platform ends up with two kinds of topic on the same cluster, configured differently on purpose.

payments (source of truth)clickstream (analytics feed)
Keyorder idsession id
Replication factor3, one replica per zone3
min.insync.replicas21
Producer acksall1
Unclean leader electionoffsometimes on
One broker downWrites continue, ISR is 2Writes continue
Two brokers downWrites rejected, nothing acknowledged is lost, reads continue up to the high watermarkWrites continue on one replica; they are lost if that broker fails
What the producing service must doKeep the event (for example in an outbox table) and retry laterNothing; a gap in analytics is accepted

For payments, a rolling restart of the brokers is uneventful: each partition's ISR goes from three to two and back as each broker restarts, and writes never stop. The interesting moment is an unplanned failure during that restart. With two replicas of a partition unavailable, the payment service starts getting NotEnoughReplicasException after its retries run out. That is the system working as designed: it is refusing to record a payment event on a single disk. The service has to be built for it, which is why the transactional outbox is usually where these events start.

The internal topics follow the durable pattern out of the box: the transaction state log defaults to replication factor 3 with a minimum ISR of 2, and the consumer offsets topic to replication factor 3.

Java and Spring Boot

The durability settings live on the topic, the acknowledgement setting on the producer, and the application's only real job is to treat a failed send as a failed send. The publisher below is called by an outbox relay, which marks a row as sent only when the returned future completes normally.

src/main/java/com/example/payments/kafka/PaymentTopicConfig.javaJava
package com.example.payments.kafka;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.common.config.TopicConfig;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;@Configurationpublic class PaymentTopicConfig {    public static final String PAYMENTS = "payments";    @Bean    public NewTopic paymentsTopic() {        return TopicBuilder.name(PAYMENTS)                .partitions(12)                .replicas(3)                .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2")                .config(TopicConfig.UNCLEAN_LEADER_ELECTION_ENABLE_CONFIG, "false")                .build();    }}

Why it is written this way

  • Replication factor 3 with a minimum of 2 means an acknowledged record is on two brokers, and one broker can be down without blocking writes.
  • The unclean election setting is already false by default. Writing it on the topic protects it from a cluster-wide default that someone changes later.
  • By default KafkaAdmin does not alter the configuration of a topic that already exists (spring.kafka.admin.modify-topic-configs is false). Change an existing topic with kafka-configs.sh.

Configuration

Five settings decide what an acknowledgement is worth and what happens when replicas fall away. Defaults are from the Kafka 4.4 configuration reference.

min.insync.replicasDefault 1
What it does
Topic setting (with a cluster-wide default). Minimum ISR size, leader included, for an acks=all write to be accepted. In Kafka 4.x the high watermark also does not advance below it.
When to change it
Set it to 2 with replication factor 3 for data that must not be lost. Leave it at 1 only for topics where availability matters more than any single record. Never set it equal to the replication factor.
acksDefault all
What it does
Producer setting. 0: do not wait. 1: wait for the leader's local write. all (or -1): wait for every current in-sync replica.
When to change it
Keep all for business events. 1 trades durability for lower latency, and min.insync.replicas does not protect a producer that uses it.
replica.lag.time.max.msDefault 30000 (30 seconds)
What it does
Broker setting. A follower that has not sent a fetch request, or has not caught up to the leader's log end offset, for this long is removed from the ISR.
When to change it
Lowering it shortens the stall caused by a stuck follower and makes followers drop out more easily under load or long pauses. Change it only with evidence from ISR shrink metrics.
unclean.leader.election.enableDefault false
What it does
Topic setting (with a cluster-wide default). Allow a replica outside the ISR to become leader as a last resort, accepting the loss of records it does not have.
When to change it
Enable it only per topic, where being offline is worse than losing data. When switched on dynamically in KRaft mode it is applied by a periodic check (every 5 minutes by default); kafka-leader-election.sh --election-type unclean triggers it at once.
request.timeout.msDefault 30000 (30 seconds)
What it does
Producer setting. How long the client waits for a response before resending the request.
When to change it
The documentation advises keeping it larger than replica.lag.time.max.ms, so that an ISR shrink completes before the producer gives up on the request and resends.
delivery.timeout.msDefault 120000 (2 minutes)
What it does
Producer setting. Total time a record may spend being batched, sent and retried before the send is reported as failed.
When to change it
This is how long a producer waits out an ISR that is below the minimum. Make sure whatever calls send() can tolerate that wait or does not block on it.
eligible.leader.replicas.versionDefault 1 on clusters created with 4.1 or later; 0 on 4.0 and on upgraded clusters until set
What it does
Cluster feature. When 1, the controller tracks replicas that left the ISR while it was below the minimum and may elect them safely.
When to change it
Enable it after upgrading an older cluster. Once on, min.insync.replicas must be managed at cluster or topic level, not in an individual broker's configuration.

Hands-on exercise

Shrink the ISR until writes are refused

Take replicas of one partition away one at a time and, at each step, predict before you look whether an acks=all producer is acknowledged and how far a consumer can read.

Steps · 0 of 7 done

payments with min.insync.replicas = 2

Simulation · not a real cluster

Stop and restart brokers and watch the ISR, the high watermark and what a producer with acks=all is told at each step.

Take it further in the lab: Broker failure and leader election

What can go wrong

acks=all with min.insync.replicas left at 1
Symptom
After a second broker failure, records that producers were told had succeeded are missing, although the topic has replication factor 3 and every producer uses acks=all.
Cause
The default minimum is 1. Once the ISR had shrunk to the leader, acks=all was satisfied by the leader alone, and those records were on one disk when it failed.
Fix
Set min.insync.replicas=2 on the topic, or as the cluster default, for every topic that uses acks=all for durability.
min.insync.replicas equal to the replication factor
Symptom
Every rolling restart or single broker failure produces NotEnoughReplicasException for the partitions hosted on that broker.
Cause
With replication factor 3 and a minimum of 3, any absent replica takes the ISR below the minimum.
Fix
Keep the minimum at least one below the replication factor: 3 and 2 is the usual pair. If two copies must always exist during maintenance, raise the replication factor instead.
A producer that does not use acks=all
Symptom
The topic is configured for durability, yet an audit after a broker failure finds acknowledged records missing from one service. In Kafka 4.x the same service's consumers also stop receiving new records while the ISR is below the minimum, although its producer reports success.
Cause
That producer runs with acks=1 (an old configuration or a library default). min.insync.replicas only rejects writes that ask for acks=all, and it now also holds the high watermark.
Fix
Set acks=all on every producer of the topic. If a producer truly wants acks=1, give it a topic whose minimum ISR is 1.
An ISR that keeps shrinking and expanding
Symptom
Produce latency spikes for seconds at a time with no broker down. IsrShrinksPerSec and IsrExpandsPerSec are both regularly above zero.
Cause
A follower repeatedly fails to keep up and then recovers: a slow or failing disk, long garbage-collection pauses, a saturated network link, or too few replica fetcher threads for the write rate.
Fix
Find the broker that is always the one leaving and fix its resource problem. Raising replica.lag.time.max.ms hides the symptom and lengthens every stall.
Enabling unclean election to end an outage
Symptom
An offline partition comes back after unclean election is switched on. Later, consumers report offsets out of range, and downstream systems hold records that are no longer in the topic.
Cause
The replica that became leader was missing committed records. Its shorter log became the truth and the other replicas truncated to match.
Fix
Treat it as a decision to lose data, made per topic and in advance. For topics that are a source of truth, wait for an in-sync or eligible replica, and prefer more replicas spread across zones so that the situation is rare.

How to troubleshoot

Producers fail with NotEnoughReplicasException

Check

kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --under-min-isr-partitions
How to read it
Lists partitions whose Isr has fewer members than min.insync.replicas. Compare Replicas with Isr to see which brokers are missing. Writes resume when enough of them have caught up; nothing acknowledged has been lost.
Need an early warning before writes are refused

Check

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions, name=AtMinIsrPartitionCount and name=UnderMinIsrPartitionCount
How to read it
All three should be 0. Under-replicated means some replica is out of sync. At-min-ISR means one more failure will stop acks=all writes. Under-min-ISR means it already has.
Produce latency jumps for about 30 seconds, then recovers

Check

kafka.server:type=ReplicaManager,name=IsrShrinksPerSec, and kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica on each follower
How to read it
A shrink at the end of the stall confirms the leader was waiting for a follower that had stopped keeping up. Apart from broker restarts, the expected shrink and expand rates are 0.
Consumers receive nothing new although producers report success

Check

kafka-topics.sh --describe --under-min-isr-partitions, and the `acks` setting of the producer
How to read it
On Kafka 4.x the high watermark does not advance while the ISR is below the minimum. Records written with acks=1 or acks=0 sit above it, invisible to consumers, until the ISR recovers.
A partition shows Leader: none

Check

kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --unavailable-partitions
How to read it
No in-sync replica is available. The Isr column shows the last in-sync set and, where the feature is on, Elr shows replicas that are still safe to elect. Bringing one of those brokers back restores the partition without loss. Forcing kafka-leader-election.sh --election-type unclean --topic payments --partition 0 restores it with whatever the chosen replica has.
Suspect that data was lost in an election

Check

kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec
How to read it
Should always be 0. Any other value means a replica outside the ISR was made leader and committed records may have been discarded.

Key takeaways

  • In sync means: the broker has a live session with the controller, and the follower has caught up to the leader's log end within replica.lag.time.max.ms. It is a time limit, not a record count.
  • The high watermark is the lowest log end offset in the ISR. Consumers read only below it, and a new leader chosen from the ISR has everything below it.
  • acks=all waits for the ISR as it is right now. It is a durability guarantee only with min.insync.replicas of at least 2 and a replication factor above that.
  • Below the minimum ISR, acks=all writes fail with NotEnoughReplicas and reads continue up to the high watermark. That is the system choosing durability over availability.
  • A stuck follower stalls acknowledgements for up to replica.lag.time.max.ms before it is removed.
  • Unclean leader election brings a partition back by discarding records. It is off by default and should stay off for anything that is a source of truth.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

kafka-topics.sh --describe shows payments-2 Leader: 1 Replicas: 1,2,3 Isr: 1 and the topic has min.insync.replicas=2. The leader's log end offset and high watermark are both 5,040. The payment service produces with acks=all and default timeouts. A consumer group has committed offset 5,000 for this partition.

What do the producer and the consumer experience over the next few minutes?

Scenario 2 of 3

events-0 has Replicas: 4,5,6, replication factor 3, and min.insync.replicas was never set. Brokers 4 and 6 have been down for an hour, so Isr: 5. Producers use acks=all and have reported only successes during that hour. Now broker 5's disk fails and cannot be recovered. Unclean leader election is disabled.

What is the state of the records acknowledged during the last hour?

Scenario 3 of 3

orders-7 has Isr: 1,2,3 with broker 1 leading and min.insync.replicas=2. Broker 3's disk controller degrades: the broker stays up and heartbeats normally, but its follower now falls steadily further behind the leader. The order service produces with acks=all and normally sees 15 ms acknowledgements. Default broker settings.

What will the order service see?

References

References: Apache Kafka documentation: Design, Replication (ISR, unclean leader election, availability and durability) ↗ · Apache Kafka documentation: Broker configs (min.insync.replicas, replica.lag.time.max.ms) ↗ · Apache Kafka documentation: Topic configs ↗ · Apache Kafka documentation: Producer configs (acks, request.timeout.ms, delivery.timeout.ms) ↗ · Apache Kafka documentation: Eligible Leader Replicas ↗ · Apache Kafka documentation: Monitoring (ISR and replica metrics) ↗ · KIP-966: Eligible Leader Replicas ↗ · KIP-497: Add inter-broker API to alter ISR ↗ · Spring for Apache Kafka reference: Configuring topics ↗ · Report a problem