Path 2 · lesson 8 of 8
Partition leadership and leader election
Who picks a new partition leader when a broker fails, why leader epochs keep replicas consistent, what unclean election costs, and what clients do meanwhile.
What you will learn
- Tell partition leader election apart from controller election and say who decides each
- Name the events that move a partition's leadership and how long each leaves the partition without a leader
- Explain how leader epochs fence a stale leader and make replicas truncate to a consistent log
- Decide for a given topic whether unclean leader election is acceptable, and state exactly what it can lose
- Configure a producer so that a leader election causes a delay instead of an error, a duplicate or a reordering
- Read
kafka-topics.sh --describeoutput during a failure and act on it
Before you start
The concept
Every partition has one leader replica at a time. Producers write to it, followers copy from it, and by default consumers read from it. When the broker hosting the leader stops, the partition cannot accept writes until another replica takes over. Choosing that replica is partition leader election.
The name suggests replicas voting. They do not. The active controller chooses the new leader, picking a replica from the partition's in-sync replica set (ISR), and writes the decision to the metadata log. Brokers learn of it by replaying that log. The ISR is what makes this safe: a record counts as committed only once every ISR member has it, so any ISR member has every committed record and can lead without losing one.
Controller election (previous lesson)
- Chooses the active controller of the cluster
- Decided by a majority vote among the controller quorum (Raft)
- Happens once per controller failure, for the whole cluster
- Protects the metadata log,
__cluster_metadata - Needs a majority of controllers alive
Partition leader election (this lesson)
- Chooses the leader replica of one topic partition
- Decided by the active controller alone: no voting among replicas
- Happens per partition, thousands of times when a broker fails
- Protects your records in that partition
- Needs one eligible replica alive, and a working controller to choose it
Stop a leader and watch the election
Simulation · not a real clusterPick a partition, note its leader and ISR, then stop the leader's broker. Watch which replica the controller promotes, how the ISR changes, and what happens when the broker comes back.
Why it exists
A single leader per partition is what gives a partition one order: one broker assigns offsets, so there is never a dispute about which record came first. The price is that the leader is a single point of failure for writes, so replacing it must be automatic, fast and safe. Three properties are needed, and each rules out a simpler design.
- Never two leaders. If two brokers both accepted writes for the same partition, the same offset would hold different records on different brokers. So the decision has to be made in one place and stamped with a number, the leader epoch, that lets every broker reject a leader that has been replaced.
- Never lose a committed record. The new leader must already hold everything that was acknowledged to producers using
acks=all. Choosing from the ISR guarantees that without comparing logs at election time. - Be quick for thousands of partitions. A broker can lead thousands of partitions. If each partition ran its own vote, a broker failure would start thousands of elections at once. The Kafka design documentation gives this as the reason for a central controller: it decides for all affected partitions and batches the changes.
The alternative, majority voting per partition, would also need more copies for the same fault tolerance. A majority scheme needs 3 replicas to survive one failure and 5 to survive two. With an ISR, f+1 replicas survive f failures without losing committed data, which matters when the thing being replicated is all of your data rather than a small metadata log. Kafka uses majority voting where it is cheap (the controller quorum) and the ISR where it is not (partitions).
How it works inside
What triggers a leadership change
| Trigger | What happens | Time without a leader |
|---|---|---|
| Broker crash or network loss | The controller stops receiving the broker's heartbeats. After broker.session.timeout.ms it fences the broker, removes it from every ISR and elects new leaders for the partitions it led. | Up to the session timeout (9 seconds by default) to detect, then a very short election. |
| Controlled shutdown (SIGTERM, rolling restart) | The broker tells the controller in its heartbeat that it wants to stop. The controller moves its leaderships to other ISR members first, then lets it shut down. | Per the documentation, a few milliseconds per partition. |
| Preferred leader election | The controller moves leadership back to the first replica in the partition's replica list, if that replica is in the ISR. Automatic by default, or run by hand. | Brief: the same kind of clean hand-over. |
| Partition reassignment | When a reassignment removes the current leader from the replica set, leadership moves to a replica that stays, once the new replicas are in sync. | Brief: both old and new leader are in sync. |
| Manual unclean election | An operator forces a replica outside the ISR to become leader for a partition that has no leader. | The partition was already offline; this ends the outage at the cost of data. |
How the controller chooses
For each partition that needs a leader, the controller looks at the replica list and the current ISR, both of which it holds in memory, and picks a replica that is in the ISR and on an unfenced broker. It writes a partition change record with the new leader, the reduced ISR and a leader epoch one higher than before. Once the quorum has committed that record, brokers replay it: the chosen broker starts acting as leader and the other replicas start fetching from it. No log comparison is needed, because ISR membership already says the replica is complete up to the high watermark.
If no replica is eligible (every ISR member is down), the partition has no leader and is offline: producers cannot write and consumers cannot read until an eligible replica returns. This is deliberate. With unclean.leader.election.enable=false, the default, Kafka prefers to be unavailable over promoting a replica that is known to be missing committed records.
Leader epochs
The leader epoch is a counter per partition that the controller increases with every leader change. The leader stamps it on every record batch it appends, and each replica keeps a small file, leader-epoch-checkpoint, listing the offset at which each epoch began. That gives every stretch of the log an owner, and it is used for two things.
- Fencing a stale leader. Requests between brokers carry the sender's view of the leader epoch. A broker that was paused, or cut off from the controller, and still believes it leads with epoch 7 finds its requests refused once the rest of the cluster is on epoch 8 (the error is named FENCED_LEADER_EPOCH). It cannot gather acknowledgements from followers, so nothing it accepts with
acks=allis ever confirmed. - Truncating to a consistent log. A replica that was leader, or was simply ahead, may hold records at the end of its log that were never committed and that the new leader does not have. When it rejoins as a follower it asks the new leader where the epoch of its last records ended, removes everything it has beyond that offset, and then fetches normally.
Before leader epochs (KIP-101), a restarting follower truncated its log to its own high watermark. A follower learns the high watermark one fetch later than the leader, so it could cut off a record that was in fact committed, and if it then became leader that record was gone for good. In another case, after several hard failures two replicas could end up with different records at the same offset and stay that way. Epochs fix both: the follower truncates only what the leader's history says is not part of the log, and the result is the same sequence of records on every replica.
Preferred leaders
The first broker in a partition's replica list is its preferred leader. Replica lists are assigned so that preferred leaders are spread evenly. After a failure, the surviving brokers lead more partitions than their share, and a broker that restarts comes back leading nothing. With auto.leader.rebalance.enable=true (the default), the controller checks every leader.imbalance.check.interval.seconds (300) and moves leadership back to preferred replicas that are in the ISR. You can trigger the same thing at once with kafka-leader-election.sh --election-type preferred.
Unclean leader election
Unclean election means promoting a replica that is not in the ISR (or ELR) because nothing better is alive. That replica's log becomes the truth. Every record the old leader had beyond it is lost, including records that were committed and acknowledged with acks=all, and when the other replicas return they truncate to match the new leader. Consumers can be affected too: a group that had already read past the new end of the log holds a position that no longer exists or now points at different records.
unclean.leader.election.enable=false (default)
- With no eligible replica, the partition stays offline
- No committed record is lost
- Recovery requires an in-sync or eligible replica to come back
- Right for orders, payments, inventory, anything you cannot recreate
unclean.leader.election.enable=true
- An out-of-sync replica may become leader
- The partition is writable again quickly
- Records the new leader lacks are lost for good
- Only defensible for data where a gap is cheaper than an outage, such as metrics
It is a topic-level setting, so the choice can differ per topic. There is also a one-off form that leaves the configuration alone: kafka-leader-election.sh --election-type unclean for a named partition, added by KIP-460 so that an operator can end a specific outage without leaving the risk switched on. The reference notes that when the setting is enabled dynamically, the controller acts on it from a periodic task (every 5 minutes by default), so the command is the way to trigger it immediately.
What clients do during an election
Clients cache which broker leads each partition. When leadership moves, a producer's next request to the old leader fails with a retriable error (NOT_LEADER_OR_FOLLOWER if that broker is up, a disconnect or timeout if it is not; LEADER_NOT_AVAILABLE while no leader exists). The client requests fresh metadata from any broker, waits retry.backoff.ms between attempts, and resends to the new leader. All of this happens inside send(), bounded by delivery.timeout.ms (2 minutes by default). Your code sees an error only if the partition has no leader for longer than that.
- Producer retries and duplicates. If the old leader appended a batch and replicated it but failed before responding, the retry goes to the new leader, which already has the batch. With
enable.idempotence=true(the default) the new leader recognises the producer id and sequence number and does not append it twice, and ordering within the partition is kept. This covers the producer's own retries only. It does not make your consumers idempotent. - Buffering. While a partition has no leader, records for it accumulate in the producer's buffer. If the outage is long,
send()blocks for up tomax.block.mswhen the buffer is full and batches expire with a timeout exception. - Consumers. A fetch to the old leader fails, the consumer refreshes metadata and continues from the same position on the new leader. No offsets are lost, because committed offsets live in
__consumer_offsets, and no rebalance is needed, because group membership has not changed. - Coordinators move too.
__consumer_offsetsand the transaction state topic are ordinary partitions. If the failed broker led one of them, the groups and transactional producers it coordinated must find their new coordinator, which adds a short pause to commits.
Architecture
Drawing diagram…
A state diagram for one partition. From a healthy state with leader on broker 101 at epoch 7, a controlled shutdown leads directly to a new leader on broker 102 at epoch 8, while a crash first passes through a state where the leader is not heartbeating. If no in-sync replica is alive the partition is offline until an eligible replica returns or an unclean election promotes an out-of-sync replica with data loss. When broker 101 restarts it truncates and catches up as a follower, and a preferred election later moves leadership back to it at epoch 9.
stateDiagram-v2 state "Leader on broker 101, epoch 7, ISR 101 102 103" as Healthy state "Leader 101 not heartbeating" as Suspect state "Leader on broker 102, epoch 8, ISR 102 103" as Elected state "Offline, no eligible replica" as Offline state "Leader from outside the ISR, records lost" as Unclean state "Broker 101 truncates and catches up as follower" as Rejoin state "Leader back on preferred broker 101, epoch 9" as Preferred [*] --> Healthy Healthy --> Elected : controlled shutdown of 101 Healthy --> Suspect : broker 101 crashes Suspect --> Elected : session timeout, controller picks an ISR member Suspect --> Offline : no ISR member is alive Offline --> Elected : an eligible replica returns Offline --> Unclean : unclean election enabled or forced Elected --> Rejoin : broker 101 restarts Rejoin --> Preferred : 101 back in ISR, preferred election Preferred --> Healthy : ISR complete again
- The two paths out of the healthy state differ only in detection time. A controlled shutdown skips the wait, which is why rolling restarts should always use SIGTERM and never a hard kill.
- Each move to a new leader increases the epoch: 7, then 8, then 9. The epoch, not the broker id, is what identifies a period of leadership.
- Offline is a safe state, not a broken one: the partition is waiting for a replica that has all committed records. The unclean path out of it is the only arrow in the picture that loses data.
- The returning broker is a follower first. It gets leadership back only after it has caught up and rejoined the ISR.
Message flow
Drawing diagram…
A sequence diagram. A producer writes to broker 101, the leader, and is acknowledged. Broker 101 crashes and the next batch gets no response. After the session timeout the active controller fences 101 and records broker 102 as leader with epoch 8. Brokers 102 and 103 fetch the change, 102 becomes leader and 103 follows it. The producer refreshes metadata, retries the batch against 102 with the same sequence number and is acknowledged. Broker 101 later restarts, truncates and follows 102.
sequenceDiagram participant P as Producer participant B1 as Broker 101, old leader participant C as Active controller participant B2 as Broker 102 participant B3 as Broker 103 P->>B1: Produce to orders.events-0 with acks=all B1-->>P: Ack at offset 880 Note over B1: Broker 101 crashes P->>B1: Produce next batch Note over P: No response, the batch stays buffered and is retried Note over C: No heartbeat from 101 for broker.session.timeout.ms C->>C: Fence 101, set leader 102, ISR 102 103, epoch 8 B2->>C: Fetch metadata C-->>B2: Partition change record B3->>C: Fetch metadata C-->>B3: Partition change record Note over B2: Becomes leader at epoch 8 B3->>B2: Fetch as follower with epoch 8 P->>B2: Metadata request B2-->>P: Leader of orders.events-0 is 102 P->>B2: Retry the batch with the same sequence number B2-->>P: Ack at offset 881 Note over B1: Restarts later, truncates to the end of epoch 7, follows 102
- The pause the producer sees is almost entirely detection: up to
broker.session.timeout.msbefore the controller acts. The election itself is one metadata record. - Brokers 102 and 103 are not told individually. They fetch the metadata log as usual and act on the partition change record when they replay it.
- The producer learns the new leader from any broker's metadata cache, then resends. The same sequence number lets the new leader detect a batch it already has.
- Offset 880 was acknowledged with
acks=all, so 102 and 103 both had it before 101 failed. Nothing committed is lost, and when 101 returns it removes any uncommitted records it holds beyond what epoch 7 contained on the new leader.
In a real system
A typical order platform: orders.events is keyed by order id, has 12 partitions, replication factor 3 across three availability zones, min.insync.replicas=2, and unclean election disabled. The order service produces with acks=all and idempotence. Brokers are restarted one at a time for an upgrade every few weeks, and occasionally a host fails.
| Event | What the cluster does | What the order service sees |
|---|---|---|
| Rolling restart of one broker (SIGTERM) | Controlled shutdown moves its 4 leaderships to in-sync replicas before it stops. ISR is 2 of 3 until it returns and catches up, which still satisfies min.insync.replicas=2. | A few retried requests and a small latency bump. No errors and no duplicates. |
| One broker's host dies | About 9 seconds until the controller fences it, then new leaders for its 4 partitions. The other 8 partitions are unaffected. | Sends to those 4 partitions stall for some seconds, then complete. Well within the 2 minute delivery timeout. |
| A second broker dies before the first is back | Partitions that had both replicas there are down to 1 in-sync replica. They still have a leader, but acks=all writes are rejected because the ISR is below min.insync.replicas. | Not-enough-replicas errors that are retried until the delivery timeout, then failures. Reads still work. |
| All three replicas of a partition are down | The partition is offline. It gets a leader again when an eligible replica restarts. | Sends for keys on that partition time out. The service must keep or reject those orders, not drop them. |
The same company's metrics.host-cpu topic makes the opposite choice: replication factor 2, unclean.leader.election.enable=true. A dashboard with a gap of a few seconds is acceptable, and a monitoring topic that is offline during an incident is not. The setting is per topic precisely so that both decisions can live in one cluster.
For the rolling restart to be invisible, three things must hold together: replication factor 3, min.insync.replicas=2 (not 3, or every restart blocks writes), and one broker at a time, waiting for under-replicated partitions to return to zero before the next.
Java and Spring Boot
Application code never takes part in an election. Its job is to be configured so that an election shows up as a delay. Below: a producer set up for that, the topic settings that decide the trade-off, an Admin-based check of leaders and ISR, and the commands an operator uses.
spring: kafka: bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:broker101:9092,broker102:9092,broker103:9092} producer: acks: all key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: # Retries during an election must not duplicate or reorder (default: true). enable.idempotence: true # Upper bound on send + retries. Must comfortably exceed the time to # detect a dead broker (broker.session.timeout.ms, 9 s) plus the election. delivery.timeout.ms: 120000 request.timeout.ms: 30000 retry.backoff.ms: 100 retry.backoff.max.ms: 1000 consumer: group-id: payment-service enable-auto-commit: falseWhy it is written this way
- These are the client defaults in current Kafka versions, written out so nobody lowers them by accident. A
delivery.timeout.msof a few seconds turns every broker failure into application errors. - Several brokers are listed in
bootstrap-serversso the client can still fetch metadata when one of them is the broker that just failed. acks: allonly gives durability across a leader change together with the topic'smin.insync.replicasand replication factor, set below.- Consumers need no special configuration for leader changes: the client refreshes metadata and continues from its position.
package com.example.orders;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.common.config.TopicConfig;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;@Configurationpublic class OrderTopicConfig { @Bean NewTopic orderEvents() { return TopicBuilder.name("orders.events") .partitions(12) .replicas(3) // One replica may be down (restart or failure) and writes continue. .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2") // Stay offline rather than promote a replica that is missing committed orders. .config(TopicConfig.UNCLEAN_LEADER_ELECTION_ENABLE_CONFIG, "false") .build(); }}Why it is written this way
falseis already the default for unclean election. Stating it in the topic definition records the decision and protects the topic if someone changes the broker-wide default.- Replication factor 3 with
min.insync.replicas=2is the combination that survives one broker being down without blocking writes or risking committed data.
package com.example.ops;import java.util.List;import java.util.Map;import java.util.Optional;import java.util.Set;import java.util.concurrent.TimeUnit;import org.apache.kafka.clients.admin.Admin;import org.apache.kafka.clients.admin.TopicDescription;import org.apache.kafka.common.ElectionType;import org.apache.kafka.common.Node;import org.apache.kafka.common.TopicPartition;import org.springframework.kafka.core.KafkaAdmin;import org.springframework.stereotype.Component;@Componentpublic class PartitionLeadershipInspector { public record PartitionState(int partition, Integer leader, int preferredLeader, List<Integer> replicas, List<Integer> isr) { public boolean offline() { return leader == null; } public boolean underReplicated() { return isr.size() < replicas.size(); } public boolean onPreferredLeader() { return leader != null && leader == preferredLeader; } } private final KafkaAdmin kafkaAdmin; public PartitionLeadershipInspector(KafkaAdmin kafkaAdmin) { this.kafkaAdmin = kafkaAdmin; } public List<PartitionState> describe(String topic) throws Exception { try (Admin admin = Admin.create(kafkaAdmin.getConfigurationProperties())) { TopicDescription description = admin.describeTopics(List.of(topic)) .allTopicNames() .get(10, TimeUnit.SECONDS) .get(topic); return description.partitions().stream() .map(info -> new PartitionState( info.partition(), info.leader() == null ? null : info.leader().id(), info.replicas().get(0).id(), info.replicas().stream().map(Node::id).toList(), info.isr().stream().map(Node::id).toList())) .toList(); } } /** Ask the controller to move leadership back to the preferred replica. */ public Map<TopicPartition, Optional<Throwable>> electPreferredLeader(String topic, int partition) throws Exception { try (Admin admin = Admin.create(kafkaAdmin.getConfigurationProperties())) { return admin.electLeaders(ElectionType.PREFERRED, Set.of(new TopicPartition(topic, partition))) .partitions() .get(30, TimeUnit.SECONDS); } }}Why it is written this way
TopicPartitionInfo.leader()returns null when the partition has no leader. That is the offline state, and the code treats it as a value, not an exception.- The first entry of
replicas()is the preferred leader. Comparing it with the current leader shows how unbalanced leadership is after a failure. electLeadersonly asks. The controller performs the election, and only if the preferred replica is in the ISR. The returned map holds an error per partition when it could not.ElectionType.UNCLEANexists in the same API. Keep it out of application code: forcing data loss should be a deliberate operator action.
# Leader, replicas and ISR (and Elr / LastKnownElr on Kafka 4) for every partition of a topicbin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders.events# Only the partitions that need attention, across the clusterbin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --unavailable-partitionsbin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-min-isr-partitionsbin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions# Move leadership back to preferred replicas after brokers have returnedbin/kafka-leader-election.sh --bootstrap-server localhost:9092 \ --election-type preferred --all-topic-partitions# LAST RESORT, loses data: force a leader for one offline partition from outside the ISRbin/kafka-leader-election.sh --bootstrap-server localhost:9092 \ --election-type unclean --topic metrics.host-cpu --partition 3Why it is written this way
--unavailable-partitionslists partitions whose leader is not available. An empty result is what you want.- The tool performs a preferred election only where the current leader is not the preferred one, and an unclean election only where the partition has no leader.
- Run the unclean form for one named partition, after deciding that waiting for the in-sync replica is worse than losing its unreplicated records.
Configuration
Broker and topic settings that govern elections, then the client settings that decide how an election is experienced. Defaults are from the Kafka 4.4 configuration reference.
unclean.leader.election.enableDefault false
- What it does
- Whether a replica outside the ISR may be elected leader as a last resort, at the risk of losing data. Broker-wide default and topic-level override.
- When to change it
- Leave it false for business data. Enable it per topic only where a gap is cheaper than an outage. Prefer the one-off
kafka-leader-election.sh --election-type uncleanto leaving it on.
min.insync.replicasDefault 1
- What it does
- Minimum ISR size for an
acks=allwrite to be accepted. Also what makes eligible leader replicas possible, since the high watermark stops below it. - When to change it
- Set 2 with replication factor 3. With 1, a lone leader accepts writes that an election can then lose if that leader fails.
auto.leader.rebalance.enableDefault true
- What it does
- A background task in the controller moves leadership back to preferred replicas when leadership is unbalanced.
- When to change it
- Keep it on. Turn it off only if you want to control the timing yourself, and then run preferred elections after every broker restart.
leader.imbalance.check.interval.secondsDefault 300
- What it does
- How often the controller runs the leader balance check.
- When to change it
- Explains why a restarted broker leads nothing for some minutes. Run a preferred election by hand if you need balance sooner.
broker.session.timeout.msDefault 9000 (9 seconds)
- What it does
- Time without a heartbeat after which the controller fences a broker and elects new leaders for its partitions.
- When to change it
- This is most of the unavailability after a crash. Lowering it shortens failover but makes long garbage collection or network pauses look like failures.
replica.lag.time.max.msDefault 30000 (30 seconds)
- What it does
- A follower that has not caught up to the leader's log end for this long is removed from the ISR, and so from the set of clean election candidates.
- When to change it
- Rarely changed. A lower value keeps the ISR strict but shrinks it more often under load.
controlled.shutdown.enableDefault true
- What it does
- On a graceful stop, the broker has its leaderships moved away before it shuts down.
- When to change it
- Leave on, and stop brokers with SIGTERM and enough time to finish. It only succeeds if the broker's partitions have other live replicas.
delivery.timeout.ms (producer)Default 120000 (2 minutes)
- What it does
- Upper bound on the time from
send()to success or failure, including all retries. - When to change it
- Keep it well above detection plus election time. Spring Boot:
spring.kafka.producer.properties.delivery.timeout.ms.
retries (producer)Default 2147483647
- What it does
- How many times a failed request is retried. In practice
delivery.timeout.msis the limit. - When to change it
- Do not set it to 0 to avoid duplicates. Idempotence handles those, and without retries every election becomes an error.
enable.idempotence (producer)Default true
- What it does
- The broker discards a retried batch it already has, using the producer id and sequence number, and keeps per-partition order with up to 5 in-flight requests.
- When to change it
- Leave on. It requires
acks=all. It covers the producer's retries, not duplicates created by your application or by consumers.
retry.backoff.ms (clients)Default 100
- What it does
- Initial wait before retrying a failed request, growing up to
retry.backoff.max.ms(1000). - When to change it
- Leave at the default. It keeps clients from flooding brokers with metadata requests while a leader is being elected.
metadata.max.age.ms (clients)Default 300000 (5 minutes)
- What it does
- How often a client refreshes metadata when nothing forces it to. A leadership error triggers a refresh at once, regardless of this value.
- When to change it
- No need to lower it for failover. Lower it only to discover new partitions or topics sooner.
Hands-on exercise
Fail a leader, then fail the whole ISR
See that the controller picks the new leader from the ISR, that a restarted broker returns as a follower, and what the partition looks like when no in-sync replica is left. Then run the same failure with producers and consumers attached in the lab.
Steps · 0 of 6 done
Leader failure and recovery
Simulation · not a real clusterStop the broker that leads a partition, watch the controller promote an in-sync replica, then keep stopping brokers until the partition has no eligible leader.
What you should see
The new leader is always a member of the ISR at the moment of failure, and its epoch is one higher. A restarted broker is a follower until it has caught up. With one in-sync replica left the partition still has a leader and can be read, but acks=all writes are rejected because the ISR is below min.insync.replicas. With none left the partition is offline, and a broker that was out of sync is not promoted while unclean election is disabled. In the lab, the acks=all idempotent producer shows a pause and retries with no loss or duplicates, while acks=1 can lose the records that only the old leader had.
What can go wrong
Enabling unclean election to end an outage, and leaving it on
- Symptom
- After an incident, some orders that were acknowledged to the producer are missing from the topic, and a few consumers report positions beyond the end of the log.
- Cause
- An out-of-sync replica was promoted. Its log became authoritative and the committed records it lacked were truncated on the other replicas when they returned.
- Fix
- Keep
unclean.leader.election.enable=falseon business topics. If a partition must be forced, use the one-offkafka-leader-election.sh --election-type uncleanfor that partition, then reconcile the lost records from the source system.
Expecting acks=1 writes to survive a leader change
- Symptom
- A small number of records sent just before a broker failure never reach consumers. The producer logged success for them.
- Cause
acks=1is acknowledged by the leader alone. Records not yet replicated exist only there, and a clean election to another replica does not include them.- Fix
- Use
acks=allwithmin.insync.replicas=2and replication factor 3 for anything that must not be lost.
Short producer timeouts
- Symptom
- Every broker restart produces a burst of send failures (timeout exceptions) in services, although the cluster recovers within seconds.
- Cause
delivery.timeout.msorretrieswas lowered, so the producer gives up before the controller has detected the failure and elected a leader.- Fix
- Restore the defaults (
delivery.timeout.ms120000, unlimited retries) and rely on idempotence to make retries safe. Handle a real send failure explicitly, for example with an outbox.
Killing brokers during a rolling restart
- Symptom
- Each restart causes several seconds of stalled producers and a slow broker start with log recovery, instead of a barely visible blip.
- Cause
- The process was stopped with SIGKILL or the stop timeout was too short, so there was no controlled shutdown: the controller had to wait for the session timeout.
- Fix
- Stop brokers with SIGTERM and give them time. Restart one at a time and wait until under-replicated partitions are back to zero before the next.
min.insync.replicas equal to the replication factor
- Symptom
- Writes fail with not-enough-replicas errors every time a single broker is down, even for planned maintenance.
- Cause
- With replication factor 3 and
min.insync.replicas=3, losing any replica drops the ISR below the minimum. The partition has a leader but refusesacks=allwrites. - Fix
- Use
min.insync.replicasof replication factor minus one (2 of 3). This is a write-availability problem, not an election problem: check the ISR, not the leader.
Confusing a controller election with partition elections
- Symptom
- During a controller failover the team expects every partition to change leader and prepares for a traffic interruption that never comes, or the reverse.
- Cause
- The two are separate. A controller election changes who manages metadata and moves no partition leader. A broker failure moves partition leaders and involves no vote among replicas.
- Fix
- Diagnose them separately:
kafka-metadata-quorum.sh describe --statusfor the controller,kafka-topics.sh --describefor partition leaders.
How to troubleshoot
Producers time out for some keys only
Check
`bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --unavailable-partitions`, and the metric `kafka.controller:type=KafkaController,name=OfflinePartitionsCount`
- How to read it
- Listed partitions have no leader: every replica that could lead is down. The
Isrcolumn shows which broker must come back. A count above 0 is an outage for the keys that hash to those partitions.
Writes fail with not-enough-replicas errors but reads work
Check
`bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-min-isr-partitions`, and the metric `kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount`
- How to read it
- The partition has a leader, but its ISR is smaller than
min.insync.replicas. Find out why followers left the ISR (broker down, or lagging beyondreplica.lag.time.max.ms). Electing a different leader will not help.
One broker carries much more traffic than the others after a restart elsewhere
Check
`kafka.controller:type=KafkaController,name=PreferredReplicaImbalanceCount`, per-broker `kafka.server:type=ReplicaManager,name=LeaderCount`, and the `Leader` column against the first entry of `Replicas` in `kafka-topics.sh --describe`
- How to read it
- Leadership has not moved back to preferred replicas yet. The automatic check runs every 5 minutes, or run
bin/kafka-leader-election.sh --bootstrap-server localhost:9092 --election-type preferred --all-topic-partitions.
You need to know whether elections are happening and whether any was unclean
Check
Metrics `kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs` and `kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec`
- How to read it
- The election rate is non-zero when brokers fail or restart and should otherwise be flat. The unclean rate must be 0: any other value means data was lost on some partition.
Client logs fill with NOT_LEADER_OR_FOLLOWER or LEADER_NOT_AVAILABLE
Check
Whether the messages stop within seconds, and `bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --replication` if they do not
- How to read it
- A short burst is the normal sign of a leadership change: the client is refreshing metadata and retrying. If it continues, some broker is serving stale metadata or the partition really has no leader.
Under-replicated partitions stay above zero after a broker returned
Check
`bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions`, and `kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions` with `IsrExpandsPerSec`
- How to read it
- The returned broker is still catching up as a follower and is not yet an election candidate. Do not restart the next broker until this reaches zero.
Key takeaways
- The active controller elects partition leaders, from the ISR. Replicas do not vote. Controller election is a separate, majority-vote mechanism for the metadata quorum.
- Any ISR member has every committed record, so a clean election loses nothing that was acknowledged with
acks=all. - Leadership moves on broker failure (after the session timeout), controlled shutdown (almost immediately), preferred election and reassignment.
- Each new leader gets a higher leader epoch. Epochs fence stale leaders and tell returning replicas exactly what to truncate, so replicas never keep diverging logs.
- With no eligible replica the partition is offline by default. Unclean election trades committed data for availability and should be a per-topic, deliberate choice.
- Clients ride through an election by refreshing metadata and retrying within
delivery.timeout.ms. Idempotence makes those retries safe. - Records acknowledged with
acks=1can be lost in a perfectly clean election.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
Partition payments.captured-4 has replicas 101, 102, 103. Broker 103 was removed from the ISR 20 minutes ago because its disk was slow. Then a rack power failure took down 101 and 102 together. Broker 103 is running. The partition shows no leader, producers for some payment keys are timing out, and a manager asks you to run an unclean election now. Power is expected back in about 15 minutes.
What do you do?
Scenario 2 of 3
During a rolling upgrade, each broker restart causes the checkout service to log about 200 TimeoutException send failures, and each restart is followed by a 10 minute log recovery on the broker. The service sets delivery.timeout.ms=5000 and retries=1 'to fail fast'. The deployment tool stops brokers with a 5 second grace period and then kills them.
Which change removes the failures?
Scenario 3 of 3
Broker 102 had a 40 second stop-the-world pause. When it resumed, it still believed it was the leader of orders.events-7 with epoch 12. Meanwhile the controller had fenced it and made 103 the leader with epoch 13. A producer with stale metadata sends a batch to 102 with acks=all right after the pause ends.
What happens to that batch?
References
References: Apache Kafka documentation: Design, Replication (ISR, unclean leader election, replica management) ↗ · Apache Kafka documentation: Basic Kafka operations (graceful shutdown, balancing leadership) ↗ · Apache Kafka documentation: Eligible Leader Replicas ↗ · Apache Kafka documentation: Broker configuration reference ↗ · Apache Kafka documentation: Producer configuration reference ↗ · Apache Kafka documentation: Monitoring (controller and replica manager metrics) ↗ · KIP-101: Alter Replication Protocol to use Leader Epoch rather than High Watermark for Truncation ↗ · KIP-320: Allow fetchers to detect and handle log truncation ↗ · KIP-460: Admin Leader Election RPC ↗ · KIP-631: The Quorum-based Kafka Controller (broker heartbeats, fencing, controlled shutdown) ↗ · KIP-966: Eligible Leader Replicas ↗ · Apache Kafka Javadoc: Admin (describeTopics, electLeaders) ↗ · Report a problem