Path 2 · lesson 4 of 8
In-sync replicas, min.insync.replicas and the high watermark
Exactly what in sync means, how the ISR shrinks and grows, how the high watermark follows it, and what acks=all with min.insync.replicas really guarantees.
What you will learn
- State the two conditions a replica must meet to be in sync, and why lag is measured in time
- Explain how the in-sync set shrinks and grows and how the high watermark depends on it
- Work out from a partition's replica state whether a producer's write will succeed, wait or fail
- Work out from the same state what a consumer can and cannot read
- Choose replication factor,
min.insync.replicasandacksfor a topic and say what failure each choice survives - Explain what unclean leader election gives up and when a team might still enable it
Before you start
The concept
A partition's in-sync replicas (the ISR) are the replicas that currently hold a complete copy of the log. The leader is always a member. A follower is in sync when two things are true:
- Its broker has a live session with the controller, kept up by heartbeats. A broker whose heartbeats stop for
broker.session.timeout.msis removed from every ISR it was in. - It has caught up to the leader's log end offset at some point within the last
replica.lag.time.max.ms(30 seconds by default). A follower that has stopped fetching, or is fetching but never reaches the end, is removed by the leader after that time.
The second condition is about time, not about a number of records. A follower that is 50,000 records behind during a burst but reaches the end of the log every few seconds stays in sync. A follower that is only 10 records behind but has not reached the end for 30 seconds does not. There is no setting for how many records behind a follower may be.
The ISR matters because of what Kafka builds on it. A record is committed once every replica in the ISR has it. The high watermark is the offset below which every record is committed, and consumers are only ever given records below it. And when a leader fails, the controller picks the new leader from the ISR, so the new leader is guaranteed to have every committed record. Kafka's promise is exactly this: a committed record is not lost as long as at least one in-sync replica survives.
A partition and its in-sync set
Simulation · not a real clusterClick a partition to see its replicas, which of them are in sync, and where the high watermark is compared with each replica's log end offset.
Why it exists
A replicated log has to answer one question for every write: how many copies must exist before the producer is told it succeeded? Each fixed answer has a problem.
| Rule | What goes wrong |
|---|---|
| Acknowledge after the leader has written | The record exists on one disk. If the leader dies before a follower fetches it, an acknowledged record is gone. |
| Wait for every assigned replica | One slow or dead follower blocks all writes to the partition until someone intervenes. |
| Wait for a fixed majority | Safe and fast, but surviving one failure needs three copies and surviving two needs five. Expensive for bulk data. |
Kafka's answer is to wait for every replica that is currently keeping up, and to stop waiting for one that is not. The set is dynamic: a follower that falls behind is dropped from it and writes continue with the rest; when it has caught up it is added back. That gives the safety of waiting for all copies without letting one bad broker stop the partition.
The dynamic set has an obvious weakness. It can shrink all the way down to the leader alone, and at that point waiting for all in-sync replicas means waiting for nobody. min.insync.replicas exists to put a floor under it: below that size, the partition refuses writes that asked for full acknowledgement instead of quietly accepting them onto a single disk.
How it works inside
How the ISR shrinks
There are two routes. If a follower's broker dies or loses contact, the controller notices the missing heartbeats and removes that broker from the ISR of every partition it hosts. If the broker is alive but a follower is slow, the leader notices: it tracks when each follower last reached its log end offset, and when that was more than replica.lag.time.max.ms ago it asks the controller to remove the follower. Either way the change is written to the cluster metadata by the controller, so there is one authoritative ISR per partition.
Between a follower stalling and its removal, the leader is still waiting for it. Writes with acks=all to that partition are not acknowledged and the high watermark does not move, for up to 30 seconds with the default setting. A stuck follower therefore shows up as a pause in produce acknowledgements and in consumer delivery, followed by an ISR shrink, after which both resume.
How the ISR grows
A follower outside the ISR keeps fetching. When it returns after a failure it first aligns its log with the leader using the leader epoch (see replication), then copies what it is missing. Only when it has fully caught up to the leader's log end does the leader ask the controller to add it back. A replica is never re-admitted on the strength of having been in sync before: it has to re-sync completely, even if it lost unflushed data in its crash.
The high watermark
The leader knows each follower's log end offset from that follower's fetch requests. The high watermark is the lowest log end offset among the in-sync replicas: everything below it is on all of them. The leader includes the current value in its fetch responses, so followers learn it one round trip later. Since Kafka 4.0 there is a second condition: the high watermark only advances while the ISR has at least min.insync.replicas members. Earlier versions let it advance with a smaller ISR.
acks=all meets min.insync.replicas
acks=all on the producer means: do not acknowledge until every replica in the ISR at this moment has the record. min.insync.replicas on the topic means: if the ISR has fewer members than this, reject acks=all writes. For a topic with replication factor 3 and min.insync.replicas=2:
| ISR size | Producer with acks=all | Producer with acks=1 | Consumers |
|---|---|---|---|
| 3 | Acknowledged when all three replicas have the record | Acknowledged when the leader has written it | Read up to the high watermark, which keeps advancing |
| 2 | Acknowledged when both replicas have it. No margin left for another failure | As above | As above |
| 1 (leader only) | Rejected with NotEnoughReplicasException. The producer retries until delivery.timeout.ms, then the send fails | Still appended and acknowledged by the leader | Can read everything already below the high watermark. In Kafka 4.x the high watermark stops, so new acks=1 records are not delivered until the ISR recovers |
| 0 (no leader) | Fails: there is nobody to send to | Fails | Cannot fetch from this partition at all |
A related error, NotEnoughReplicasAfterAppendException, means the leader had already written the batch when the ISR dropped below the minimum. The record is in the leader's log but was not acknowledged. The producer retries it, and with idempotence enabled the retry does not create a second copy.
Durability against availability
Favour durability
- Replication factor 3,
min.insync.replicas=2, producers useacks=all unclean.leader.election.enable=false- An acknowledged record is on at least two brokers
- With two brokers down, producers get errors and must hold or reject work
- Suits payments, orders, ledgers, anything used as a source of truth
Favour availability
min.insync.replicas=1, oftenacks=1- Sometimes
unclean.leader.election.enable=true - Writes continue while any one replica is alive
- Records acknowledged by a lone leader are lost if that broker fails
- Suits metrics, clickstream, logs, where a gap is cheaper than back-pressure
Setting min.insync.replicas equal to the replication factor is rarely what anyone wants: every broker restart then takes one replica out of the ISR and stops writes to its partitions. The minimum should leave room for at least one replica to be absent.
When every in-sync replica is gone
If all members of the ISR are down, the partition has no leader. The controller can wait for one of them to return, which keeps the partition offline for as long as that takes, or it can promote a replica that was not in sync, which brings the partition back immediately with whatever that replica has. The second choice is an unclean leader election. The out-of-sync replica's log becomes the truth: committed records it never received are gone, the other replicas truncate to match it when they return, and consumers that had read further may find their offsets pointing past the end of the log. It is off by default (unclean.leader.election.enable=false), and has been since 0.11.
Kafka 4.x narrows the gap between those two choices with eligible leader replicas (KIP-966), enabled by default on clusters created with 4.1 or later. Because the high watermark no longer advances while the ISR is below the minimum, a replica that leaves the ISR at that point still holds everything up to the high watermark. The controller remembers such replicas and can elect one of them safely when the ISR is empty. Election order and preferred leaders are covered in partition leadership and election.
ISR / acks
Simulation, not a real clusterAnimated walk-through of a write under acks=all as the in-sync set shrinks.
Step 1 · The pattern
A leader and its followers
Every partition has a leader that takes writes and followers that copy them. If the leader dies, a follower that is caught up takes over.
- message
- being copied to follower
- confirmed
Architecture
Drawing diagram…
A state diagram for a follower replica. It starts catching up, becomes in sync when it reaches the leader's log end, becomes out of sync if it has not caught up within replica.lag.time.max.ms, goes offline if its broker's heartbeats are lost, and returns through catching up after a restart.
stateDiagram-v2
state "In sync" as InSync
state "Out of sync but fetching" as Lagging
state "Offline" as Offline
state "Catching up" as CatchingUp
[*] --> CatchingUp: replica assigned or broker started
CatchingUp --> InSync: reached the leader log end, leader asks controller to expand ISR
InSync --> Lagging: not caught up for replica.lag.time.max.ms, leader asks controller to shrink ISR
InSync --> Offline: broker heartbeats lost, controller removes it from ISR
Lagging --> InSync: caught up to the leader log end again
Lagging --> Offline: broker stops
Offline --> CatchingUp: broker restarts, truncates by leader epoch, fetches
note right of InSync
Counts towards acks=all
Holds back the high watermark
Eligible to become leader
end note
note right of Lagging
Not waited for
Not eligible in a clean election
end note- Membership changes in both directions go through the controller, which records the new ISR in the cluster metadata. The leader proposes; the controller decides.
- Only replicas in the In sync state are waited for by
acks=all, and only they limit the high watermark. That is why removing a slow replica lets writes flow again. - The path back in always passes through Catching up. There is no shortcut from Offline to In sync.
- A partition is under-replicated whenever at least one of its replicas is in a state other than In sync.
Message flow
Drawing diagram…
With three in-sync replicas, the leader acknowledges record A after both followers have fetched past it. Then broker 3 stalls. Record B is appended and fetched by broker 2, but the leader waits for broker 3 until thirty seconds have passed, asks the controller to shrink the ISR to brokers 1 and 2, raises the high watermark and acknowledges.
sequenceDiagram participant P as Producer acks=all participant B1 as Broker 1 leader participant B2 as Broker 2 follower participant B3 as Broker 3 follower participant K as Controller Note over B1,B3: ISR is 1 2 3 and min.insync.replicas is 2 P->>B1: Produce record A Note over B1: Appended at offset 120 B2->>B1: Fetch from 120 B3->>B1: Fetch from 120 B2->>B1: Fetch from 121 B3->>B1: Fetch from 121 Note over B1: All of the ISR has offset 120, high watermark is 121 B1-->>P: Acknowledged offset 120 Note over B3: Broker 3 disk stalls and it stops fetching P->>B1: Produce record B Note over B1: Appended at offset 121 B2->>B1: Fetch from 122 Note over B1: Waiting for broker 3, no acknowledgement yet Note over B1: 30 seconds pass without broker 3 catching up B1->>K: Shrink ISR of payments-0 to 1 2 K-->>B1: ISR is now 1 2 Note over B1: High watermark is 122 B1-->>P: Acknowledged offset 121
- The leader learns that a follower has offset 120 from that follower's next fetch, which asks for 121. Acknowledgement of record A follows the slower of the two followers.
- While broker 3 is stalled but still in the ISR, record B is on two brokers and still not acknowledged. Being in the ISR means being waited for.
- After the shrink, the ISR has two members, which meets
min.insync.replicas=2, so the write is acknowledged and consumers can now read offset 121. - If broker 2 then failed as well, the ISR would be the leader alone, below the minimum, and the next
acks=allwrite would be rejected instead of being accepted onto one disk. - The producer's
request.timeout.msis also 30 seconds by default, so during such a stall the producer may time out and resend the batch. With idempotence on, the resend is recognised and not written twice.
In a real system
A typical platform ends up with two kinds of topic on the same cluster, configured differently on purpose.
payments (source of truth) | clickstream (analytics feed) | |
|---|---|---|
| Key | order id | session id |
| Replication factor | 3, one replica per zone | 3 |
min.insync.replicas | 2 | 1 |
Producer acks | all | 1 |
| Unclean leader election | off | sometimes on |
| One broker down | Writes continue, ISR is 2 | Writes continue |
| Two brokers down | Writes rejected, nothing acknowledged is lost, reads continue up to the high watermark | Writes continue on one replica; they are lost if that broker fails |
| What the producing service must do | Keep the event (for example in an outbox table) and retry later | Nothing; a gap in analytics is accepted |
For payments, a rolling restart of the brokers is uneventful: each partition's ISR goes from three to two and back as each broker restarts, and writes never stop. The interesting moment is an unplanned failure during that restart. With two replicas of a partition unavailable, the payment service starts getting NotEnoughReplicasException after its retries run out. That is the system working as designed: it is refusing to record a payment event on a single disk. The service has to be built for it, which is why the transactional outbox is usually where these events start.
The internal topics follow the durable pattern out of the box: the transaction state log defaults to replication factor 3 with a minimum ISR of 2, and the consumer offsets topic to replication factor 3.
Java and Spring Boot
The durability settings live on the topic, the acknowledgement setting on the producer, and the application's only real job is to treat a failed send as a failed send. The publisher below is called by an outbox relay, which marks a row as sent only when the returned future completes normally.
package com.example.payments.kafka;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.common.config.TopicConfig;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;@Configurationpublic class PaymentTopicConfig { public static final String PAYMENTS = "payments"; @Bean public NewTopic paymentsTopic() { return TopicBuilder.name(PAYMENTS) .partitions(12) .replicas(3) .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2") .config(TopicConfig.UNCLEAN_LEADER_ELECTION_ENABLE_CONFIG, "false") .build(); }}Why it is written this way
- Replication factor 3 with a minimum of 2 means an acknowledged record is on two brokers, and one broker can be down without blocking writes.
- The unclean election setting is already false by default. Writing it on the topic protects it from a cluster-wide default that someone changes later.
- By default
KafkaAdmindoes not alter the configuration of a topic that already exists (spring.kafka.admin.modify-topic-configsis false). Change an existing topic withkafka-configs.sh.
spring: kafka: bootstrap-servers: kafka-a1.internal:9092,kafka-b1.internal:9092,kafka-c1.internal:9092 producer: # Wait for every in-sync replica. Together with min.insync.replicas=2 on the topic, # an acknowledged record is on at least two brokers. acks: all key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: enable.idempotence: true delivery.timeout.ms: 120000Why it is written this way
acks: allis the client default, and it is still worth stating: this is the half of the guarantee that lives in the application, and a different value here makes the topic's minimum ISR irrelevant for this producer.- Idempotence matters here because an ISR shrink often makes the producer time out and resend a batch the leader already has.
delivery.timeout.msbounds how long a send keeps retryingNotEnoughReplicasbefore the application is told it failed.
package com.example.payments.kafka;import java.util.concurrent.CompletableFuture;import org.apache.kafka.common.errors.NotEnoughReplicasAfterAppendException;import org.apache.kafka.common.errors.NotEnoughReplicasException;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.stereotype.Component;@Componentpublic class PaymentEventPublisher { public record PaymentAuthorized(String paymentId, String orderId, long amountMinor, String currency) { } private static final Logger log = LoggerFactory.getLogger(PaymentEventPublisher.class); private final KafkaTemplate<String, PaymentAuthorized> kafkaTemplate; public PaymentEventPublisher(KafkaTemplate<String, PaymentAuthorized> kafkaTemplate) { this.kafkaTemplate = kafkaTemplate; } /** * Completes normally only when the record is on every in-sync replica. * The outbox relay marks the row as sent after that, and not before. */ public CompletableFuture<Void> publish(PaymentAuthorized event) { return kafkaTemplate.send(PaymentTopicConfig.PAYMENTS, event.orderId(), event) .handle((result, failure) -> { if (failure == null) { return null; } if (causedBy(failure, NotEnoughReplicasException.class) || causedBy(failure, NotEnoughReplicasAfterAppendException.class)) { log.warn("payment {} not stored: too few in-sync replicas for {}. Leaving it in the outbox.", event.paymentId(), PaymentTopicConfig.PAYMENTS); } else { log.error("payment {} not stored", event.paymentId(), failure); } throw new PaymentEventNotStoredException(event.paymentId(), failure); }); } private static boolean causedBy(Throwable failure, Class<? extends Throwable> type) { for (Throwable t = failure; t != null; t = t.getCause()) { if (type.isInstance(t)) { return true; } } return false; } public static class PaymentEventNotStoredException extends RuntimeException { public PaymentEventNotStoredException(String paymentId, Throwable cause) { super("Payment event " + paymentId + " was not stored in Kafka", cause); } }}Why it is written this way
- Both exceptions are retriable, so the producer has already retried for
delivery.timeout.msbefore this code sees them. When that time runs out the failure may also surface as aTimeoutException; it is treated the same way here. - The failure is passed on to the caller. Logging it and returning normally would turn a refused write into a silently lost payment event.
- The cause chain is walked because Spring wraps producer failures before completing the future.
- A send that failed after
NotEnoughReplicasAfterAppendmay still be in the leader's log and may become visible later. The relay will publish the event again, so consumers must recognise a payment id they have already handled. Producer idempotence does not cover a secondsend()by the application.
# Raise the minimum ISR on an existing topic.bin/kafka-configs.sh --bootstrap-server kafka-a1.internal:9092 \ --entity-type topics --entity-name payments \ --alter --add-config min.insync.replicas=2# Confirm the override and look at the current replica state.bin/kafka-configs.sh --bootstrap-server kafka-a1.internal:9092 \ --entity-type topics --entity-name payments --describebin/kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --topic paymentsWhy it is written this way
- The setting takes effect without a restart. Check first that every partition currently has at least two in-sync replicas, or
acks=allproducers start failing at once. - On clusters with eligible leader replicas enabled, changing
min.insync.replicasfor a topic clears the recorded eligible replicas for it.
Configuration
Five settings decide what an acknowledgement is worth and what happens when replicas fall away. Defaults are from the Kafka 4.4 configuration reference.
min.insync.replicasDefault 1
- What it does
- Topic setting (with a cluster-wide default). Minimum ISR size, leader included, for an
acks=allwrite to be accepted. In Kafka 4.x the high watermark also does not advance below it. - When to change it
- Set it to 2 with replication factor 3 for data that must not be lost. Leave it at 1 only for topics where availability matters more than any single record. Never set it equal to the replication factor.
acksDefault all
- What it does
- Producer setting.
0: do not wait.1: wait for the leader's local write.all(or-1): wait for every current in-sync replica. - When to change it
- Keep
allfor business events.1trades durability for lower latency, andmin.insync.replicasdoes not protect a producer that uses it.
replica.lag.time.max.msDefault 30000 (30 seconds)
- What it does
- Broker setting. A follower that has not sent a fetch request, or has not caught up to the leader's log end offset, for this long is removed from the ISR.
- When to change it
- Lowering it shortens the stall caused by a stuck follower and makes followers drop out more easily under load or long pauses. Change it only with evidence from ISR shrink metrics.
unclean.leader.election.enableDefault false
- What it does
- Topic setting (with a cluster-wide default). Allow a replica outside the ISR to become leader as a last resort, accepting the loss of records it does not have.
- When to change it
- Enable it only per topic, where being offline is worse than losing data. When switched on dynamically in KRaft mode it is applied by a periodic check (every 5 minutes by default);
kafka-leader-election.sh --election-type uncleantriggers it at once.
request.timeout.msDefault 30000 (30 seconds)
- What it does
- Producer setting. How long the client waits for a response before resending the request.
- When to change it
- The documentation advises keeping it larger than
replica.lag.time.max.ms, so that an ISR shrink completes before the producer gives up on the request and resends.
delivery.timeout.msDefault 120000 (2 minutes)
- What it does
- Producer setting. Total time a record may spend being batched, sent and retried before the send is reported as failed.
- When to change it
- This is how long a producer waits out an ISR that is below the minimum. Make sure whatever calls
send()can tolerate that wait or does not block on it.
eligible.leader.replicas.versionDefault 1 on clusters created with 4.1 or later; 0 on 4.0 and on upgraded clusters until set
- What it does
- Cluster feature. When 1, the controller tracks replicas that left the ISR while it was below the minimum and may elect them safely.
- When to change it
- Enable it after upgrading an older cluster. Once on,
min.insync.replicasmust be managed at cluster or topic level, not in an individual broker's configuration.
Hands-on exercise
Shrink the ISR until writes are refused
Take replicas of one partition away one at a time and, at each step, predict before you look whether an acks=all producer is acknowledged and how far a consumer can read.
Steps · 0 of 7 done
payments with min.insync.replicas = 2
Simulation · not a real clusterStop and restart brokers and watch the ISR, the high watermark and what a producer with acks=all is told at each step.
What you should see
With one follower gone the ISR has two members, the partition is under-replicated, and writes are still acknowledged. With both followers gone the ISR is below the minimum: acks=all writes are refused with NotEnoughReplicas and the high watermark stops, while consumers can still read everything below it. A restarted replica is not back in the ISR until it has caught up completely; writes resume at that moment. In the last step the first broker you stopped was out of the ISR before the others stopped, so making it leader would be an unclean election.
What can go wrong
acks=all with min.insync.replicas left at 1
- Symptom
- After a second broker failure, records that producers were told had succeeded are missing, although the topic has replication factor 3 and every producer uses
acks=all. - Cause
- The default minimum is 1. Once the ISR had shrunk to the leader,
acks=allwas satisfied by the leader alone, and those records were on one disk when it failed. - Fix
- Set
min.insync.replicas=2on the topic, or as the cluster default, for every topic that usesacks=allfor durability.
min.insync.replicas equal to the replication factor
- Symptom
- Every rolling restart or single broker failure produces
NotEnoughReplicasExceptionfor the partitions hosted on that broker. - Cause
- With replication factor 3 and a minimum of 3, any absent replica takes the ISR below the minimum.
- Fix
- Keep the minimum at least one below the replication factor: 3 and 2 is the usual pair. If two copies must always exist during maintenance, raise the replication factor instead.
A producer that does not use acks=all
- Symptom
- The topic is configured for durability, yet an audit after a broker failure finds acknowledged records missing from one service. In Kafka 4.x the same service's consumers also stop receiving new records while the ISR is below the minimum, although its producer reports success.
- Cause
- That producer runs with
acks=1(an old configuration or a library default).min.insync.replicasonly rejects writes that ask foracks=all, and it now also holds the high watermark. - Fix
- Set
acks=allon every producer of the topic. If a producer truly wantsacks=1, give it a topic whose minimum ISR is 1.
An ISR that keeps shrinking and expanding
- Symptom
- Produce latency spikes for seconds at a time with no broker down.
IsrShrinksPerSecandIsrExpandsPerSecare both regularly above zero. - Cause
- A follower repeatedly fails to keep up and then recovers: a slow or failing disk, long garbage-collection pauses, a saturated network link, or too few replica fetcher threads for the write rate.
- Fix
- Find the broker that is always the one leaving and fix its resource problem. Raising
replica.lag.time.max.mshides the symptom and lengthens every stall.
Enabling unclean election to end an outage
- Symptom
- An offline partition comes back after unclean election is switched on. Later, consumers report offsets out of range, and downstream systems hold records that are no longer in the topic.
- Cause
- The replica that became leader was missing committed records. Its shorter log became the truth and the other replicas truncated to match.
- Fix
- Treat it as a decision to lose data, made per topic and in advance. For topics that are a source of truth, wait for an in-sync or eligible replica, and prefer more replicas spread across zones so that the situation is rare.
How to troubleshoot
Producers fail with NotEnoughReplicasException
Check
kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --under-min-isr-partitions
- How to read it
- Lists partitions whose
Isrhas fewer members thanmin.insync.replicas. CompareReplicaswithIsrto see which brokers are missing. Writes resume when enough of them have caught up; nothing acknowledged has been lost.
Need an early warning before writes are refused
Check
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions, name=AtMinIsrPartitionCount and name=UnderMinIsrPartitionCount
- How to read it
- All three should be 0. Under-replicated means some replica is out of sync. At-min-ISR means one more failure will stop
acks=allwrites. Under-min-ISR means it already has.
Produce latency jumps for about 30 seconds, then recovers
Check
kafka.server:type=ReplicaManager,name=IsrShrinksPerSec, and kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica on each follower
- How to read it
- A shrink at the end of the stall confirms the leader was waiting for a follower that had stopped keeping up. Apart from broker restarts, the expected shrink and expand rates are 0.
Consumers receive nothing new although producers report success
Check
kafka-topics.sh --describe --under-min-isr-partitions, and the `acks` setting of the producer
- How to read it
- On Kafka 4.x the high watermark does not advance while the ISR is below the minimum. Records written with
acks=1oracks=0sit above it, invisible to consumers, until the ISR recovers.
A partition shows Leader: none
Check
kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --unavailable-partitions
- How to read it
- No in-sync replica is available. The
Isrcolumn shows the last in-sync set and, where the feature is on,Elrshows replicas that are still safe to elect. Bringing one of those brokers back restores the partition without loss. Forcingkafka-leader-election.sh --election-type unclean --topic payments --partition 0restores it with whatever the chosen replica has.
Suspect that data was lost in an election
Check
kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec
- How to read it
- Should always be 0. Any other value means a replica outside the ISR was made leader and committed records may have been discarded.
Key takeaways
- In sync means: the broker has a live session with the controller, and the follower has caught up to the leader's log end within
replica.lag.time.max.ms. It is a time limit, not a record count. - The high watermark is the lowest log end offset in the ISR. Consumers read only below it, and a new leader chosen from the ISR has everything below it.
acks=allwaits for the ISR as it is right now. It is a durability guarantee only withmin.insync.replicasof at least 2 and a replication factor above that.- Below the minimum ISR,
acks=allwrites fail with NotEnoughReplicas and reads continue up to the high watermark. That is the system choosing durability over availability. - A stuck follower stalls acknowledgements for up to
replica.lag.time.max.msbefore it is removed. - Unclean leader election brings a partition back by discarding records. It is off by default and should stay off for anything that is a source of truth.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
kafka-topics.sh --describe shows payments-2 Leader: 1 Replicas: 1,2,3 Isr: 1 and the topic has min.insync.replicas=2. The leader's log end offset and high watermark are both 5,040. The payment service produces with acks=all and default timeouts. A consumer group has committed offset 5,000 for this partition.
What do the producer and the consumer experience over the next few minutes?
Scenario 2 of 3
events-0 has Replicas: 4,5,6, replication factor 3, and min.insync.replicas was never set. Brokers 4 and 6 have been down for an hour, so Isr: 5. Producers use acks=all and have reported only successes during that hour. Now broker 5's disk fails and cannot be recovered. Unclean leader election is disabled.
What is the state of the records acknowledged during the last hour?
Scenario 3 of 3
orders-7 has Isr: 1,2,3 with broker 1 leading and min.insync.replicas=2. Broker 3's disk controller degrades: the broker stays up and heartbeats normally, but its follower now falls steadily further behind the leader. The order service produces with acks=all and normally sees 15 ms acknowledgements. Default broker settings.
What will the order service see?
References
References: Apache Kafka documentation: Design, Replication (ISR, unclean leader election, availability and durability) ↗ · Apache Kafka documentation: Broker configs (min.insync.replicas, replica.lag.time.max.ms) ↗ · Apache Kafka documentation: Topic configs ↗ · Apache Kafka documentation: Producer configs (acks, request.timeout.ms, delivery.timeout.ms) ↗ · Apache Kafka documentation: Eligible Leader Replicas ↗ · Apache Kafka documentation: Monitoring (ISR and replica metrics) ↗ · KIP-966: Eligible Leader Replicas ↗ · KIP-497: Add inter-broker API to alter ISR ↗ · Spring for Apache Kafka reference: Configuring topics ↗ · Report a problem