Skip to content

Path 2 · lesson 8 of 8

Intermediate24 minApache Kafka 4.x, KRaft only (checked against the 4.4 documentation, KIP-101, KIP-320, KIP-460, KIP-631 and KIP-966) · Java 21 · Spring Boot 3.5 with Spring for Apache Kafka 3.3

Partition leadership and leader election

Who picks a new partition leader when a broker fails, why leader epochs keep replicas consistent, what unclean election costs, and what clients do meanwhile.

What you will learn

  • Tell partition leader election apart from controller election and say who decides each
  • Name the events that move a partition's leadership and how long each leaves the partition without a leader
  • Explain how leader epochs fence a stale leader and make replicas truncate to a consistent log
  • Decide for a given topic whether unclean leader election is acceptable, and state exactly what it can lose
  • Configure a producer so that a leader election causes a delay instead of an error, a duplicate or a reordering
  • Read kafka-topics.sh --describe output during a failure and act on it

Before you start

The concept

Every partition has one leader replica at a time. Producers write to it, followers copy from it, and by default consumers read from it. When the broker hosting the leader stops, the partition cannot accept writes until another replica takes over. Choosing that replica is partition leader election.

The name suggests replicas voting. They do not. The active controller chooses the new leader, picking a replica from the partition's in-sync replica set (ISR), and writes the decision to the metadata log. Brokers learn of it by replaying that log. The ISR is what makes this safe: a record counts as committed only once every ISR member has it, so any ISR member has every committed record and can lead without losing one.

Controller election (previous lesson)

  • Chooses the active controller of the cluster
  • Decided by a majority vote among the controller quorum (Raft)
  • Happens once per controller failure, for the whole cluster
  • Protects the metadata log, __cluster_metadata
  • Needs a majority of controllers alive

Partition leader election (this lesson)

  • Chooses the leader replica of one topic partition
  • Decided by the active controller alone: no voting among replicas
  • Happens per partition, thousands of times when a broker fails
  • Protects your records in that partition
  • Needs one eligible replica alive, and a working controller to choose it

Stop a leader and watch the election

Simulation · not a real cluster

Pick a partition, note its leader and ISR, then stop the leader's broker. Watch which replica the controller promotes, how the ISR changes, and what happens when the broker comes back.

Why it exists

A single leader per partition is what gives a partition one order: one broker assigns offsets, so there is never a dispute about which record came first. The price is that the leader is a single point of failure for writes, so replacing it must be automatic, fast and safe. Three properties are needed, and each rules out a simpler design.

  • Never two leaders. If two brokers both accepted writes for the same partition, the same offset would hold different records on different brokers. So the decision has to be made in one place and stamped with a number, the leader epoch, that lets every broker reject a leader that has been replaced.
  • Never lose a committed record. The new leader must already hold everything that was acknowledged to producers using acks=all. Choosing from the ISR guarantees that without comparing logs at election time.
  • Be quick for thousands of partitions. A broker can lead thousands of partitions. If each partition ran its own vote, a broker failure would start thousands of elections at once. The Kafka design documentation gives this as the reason for a central controller: it decides for all affected partitions and batches the changes.

The alternative, majority voting per partition, would also need more copies for the same fault tolerance. A majority scheme needs 3 replicas to survive one failure and 5 to survive two. With an ISR, f+1 replicas survive f failures without losing committed data, which matters when the thing being replicated is all of your data rather than a small metadata log. Kafka uses majority voting where it is cheap (the controller quorum) and the ISR where it is not (partitions).

How it works inside

What triggers a leadership change

TriggerWhat happensTime without a leader
Broker crash or network lossThe controller stops receiving the broker's heartbeats. After broker.session.timeout.ms it fences the broker, removes it from every ISR and elects new leaders for the partitions it led.Up to the session timeout (9 seconds by default) to detect, then a very short election.
Controlled shutdown (SIGTERM, rolling restart)The broker tells the controller in its heartbeat that it wants to stop. The controller moves its leaderships to other ISR members first, then lets it shut down.Per the documentation, a few milliseconds per partition.
Preferred leader electionThe controller moves leadership back to the first replica in the partition's replica list, if that replica is in the ISR. Automatic by default, or run by hand.Brief: the same kind of clean hand-over.
Partition reassignmentWhen a reassignment removes the current leader from the replica set, leadership moves to a replica that stays, once the new replicas are in sync.Brief: both old and new leader are in sync.
Manual unclean electionAn operator forces a replica outside the ISR to become leader for a partition that has no leader.The partition was already offline; this ends the outage at the cost of data.
Every one of these is decided by the active controller and recorded in the metadata log.

How the controller chooses

For each partition that needs a leader, the controller looks at the replica list and the current ISR, both of which it holds in memory, and picks a replica that is in the ISR and on an unfenced broker. It writes a partition change record with the new leader, the reduced ISR and a leader epoch one higher than before. Once the quorum has committed that record, brokers replay it: the chosen broker starts acting as leader and the other replicas start fetching from it. No log comparison is needed, because ISR membership already says the replica is complete up to the high watermark.

If no replica is eligible (every ISR member is down), the partition has no leader and is offline: producers cannot write and consumers cannot read until an eligible replica returns. This is deliberate. With unclean.leader.election.enable=false, the default, Kafka prefers to be unavailable over promoting a replica that is known to be missing committed records.

Leader epochs

The leader epoch is a counter per partition that the controller increases with every leader change. The leader stamps it on every record batch it appends, and each replica keeps a small file, leader-epoch-checkpoint, listing the offset at which each epoch began. That gives every stretch of the log an owner, and it is used for two things.

  • Fencing a stale leader. Requests between brokers carry the sender's view of the leader epoch. A broker that was paused, or cut off from the controller, and still believes it leads with epoch 7 finds its requests refused once the rest of the cluster is on epoch 8 (the error is named FENCED_LEADER_EPOCH). It cannot gather acknowledgements from followers, so nothing it accepts with acks=all is ever confirmed.
  • Truncating to a consistent log. A replica that was leader, or was simply ahead, may hold records at the end of its log that were never committed and that the new leader does not have. When it rejoins as a follower it asks the new leader where the epoch of its last records ended, removes everything it has beyond that offset, and then fetches normally.

Before leader epochs (KIP-101), a restarting follower truncated its log to its own high watermark. A follower learns the high watermark one fetch later than the leader, so it could cut off a record that was in fact committed, and if it then became leader that record was gone for good. In another case, after several hard failures two replicas could end up with different records at the same offset and stay that way. Epochs fix both: the follower truncates only what the leader's history says is not part of the log, and the result is the same sequence of records on every replica.

Preferred leaders

The first broker in a partition's replica list is its preferred leader. Replica lists are assigned so that preferred leaders are spread evenly. After a failure, the surviving brokers lead more partitions than their share, and a broker that restarts comes back leading nothing. With auto.leader.rebalance.enable=true (the default), the controller checks every leader.imbalance.check.interval.seconds (300) and moves leadership back to preferred replicas that are in the ISR. You can trigger the same thing at once with kafka-leader-election.sh --election-type preferred.

Unclean leader election

Unclean election means promoting a replica that is not in the ISR (or ELR) because nothing better is alive. That replica's log becomes the truth. Every record the old leader had beyond it is lost, including records that were committed and acknowledged with acks=all, and when the other replicas return they truncate to match the new leader. Consumers can be affected too: a group that had already read past the new end of the log holds a position that no longer exists or now points at different records.

unclean.leader.election.enable=false (default)

  • With no eligible replica, the partition stays offline
  • No committed record is lost
  • Recovery requires an in-sync or eligible replica to come back
  • Right for orders, payments, inventory, anything you cannot recreate

unclean.leader.election.enable=true

  • An out-of-sync replica may become leader
  • The partition is writable again quickly
  • Records the new leader lacks are lost for good
  • Only defensible for data where a gap is cheaper than an outage, such as metrics

It is a topic-level setting, so the choice can differ per topic. There is also a one-off form that leaves the configuration alone: kafka-leader-election.sh --election-type unclean for a named partition, added by KIP-460 so that an operator can end a specific outage without leaving the risk switched on. The reference notes that when the setting is enabled dynamically, the controller acts on it from a periodic task (every 5 minutes by default), so the command is the way to trigger it immediately.

What clients do during an election

Clients cache which broker leads each partition. When leadership moves, a producer's next request to the old leader fails with a retriable error (NOT_LEADER_OR_FOLLOWER if that broker is up, a disconnect or timeout if it is not; LEADER_NOT_AVAILABLE while no leader exists). The client requests fresh metadata from any broker, waits retry.backoff.ms between attempts, and resends to the new leader. All of this happens inside send(), bounded by delivery.timeout.ms (2 minutes by default). Your code sees an error only if the partition has no leader for longer than that.

  • Producer retries and duplicates. If the old leader appended a batch and replicated it but failed before responding, the retry goes to the new leader, which already has the batch. With enable.idempotence=true (the default) the new leader recognises the producer id and sequence number and does not append it twice, and ordering within the partition is kept. This covers the producer's own retries only. It does not make your consumers idempotent.
  • Buffering. While a partition has no leader, records for it accumulate in the producer's buffer. If the outage is long, send() blocks for up to max.block.ms when the buffer is full and batches expire with a timeout exception.
  • Consumers. A fetch to the old leader fails, the consumer refreshes metadata and continues from the same position on the new leader. No offsets are lost, because committed offsets live in __consumer_offsets, and no rebalance is needed, because group membership has not changed.
  • Coordinators move too. __consumer_offsets and the transaction state topic are ordinary partitions. If the failed broker led one of them, the groups and transactional producers it coordinated must find their new coordinator, which adds a short pause to commits.

Architecture

Drawing diagram…

A state diagram for one partition. From a healthy state with leader on broker 101 at epoch 7, a controlled shutdown leads directly to a new leader on broker 102 at epoch 8, while a crash first passes through a state where the leader is not heartbeating. If no in-sync replica is alive the partition is offline until an eligible replica returns or an unclean election promotes an out-of-sync replica with data loss. When broker 101 restarts it truncates and catches up as a follower, and a preferred election later moves leadership back to it at epoch 9.

stateDiagram-v2
  state "Leader on broker 101, epoch 7, ISR 101 102 103" as Healthy
  state "Leader 101 not heartbeating" as Suspect
  state "Leader on broker 102, epoch 8, ISR 102 103" as Elected
  state "Offline, no eligible replica" as Offline
  state "Leader from outside the ISR, records lost" as Unclean
  state "Broker 101 truncates and catches up as follower" as Rejoin
  state "Leader back on preferred broker 101, epoch 9" as Preferred
  [*] --> Healthy
  Healthy --> Elected : controlled shutdown of 101
  Healthy --> Suspect : broker 101 crashes
  Suspect --> Elected : session timeout, controller picks an ISR member
  Suspect --> Offline : no ISR member is alive
  Offline --> Elected : an eligible replica returns
  Offline --> Unclean : unclean election enabled or forced
  Elected --> Rejoin : broker 101 restarts
  Rejoin --> Preferred : 101 back in ISR, preferred election
  Preferred --> Healthy : ISR complete again
The states one partition passes through when its leader's broker stops and returns.
  • The two paths out of the healthy state differ only in detection time. A controlled shutdown skips the wait, which is why rolling restarts should always use SIGTERM and never a hard kill.
  • Each move to a new leader increases the epoch: 7, then 8, then 9. The epoch, not the broker id, is what identifies a period of leadership.
  • Offline is a safe state, not a broken one: the partition is waiting for a replica that has all committed records. The unclean path out of it is the only arrow in the picture that loses data.
  • The returning broker is a follower first. It gets leadership back only after it has caught up and rejoined the ISR.

Message flow

Drawing diagram…

A sequence diagram. A producer writes to broker 101, the leader, and is acknowledged. Broker 101 crashes and the next batch gets no response. After the session timeout the active controller fences 101 and records broker 102 as leader with epoch 8. Brokers 102 and 103 fetch the change, 102 becomes leader and 103 follows it. The producer refreshes metadata, retries the batch against 102 with the same sequence number and is acknowledged. Broker 101 later restarts, truncates and follows 102.

sequenceDiagram
  participant P as Producer
  participant B1 as Broker 101, old leader
  participant C as Active controller
  participant B2 as Broker 102
  participant B3 as Broker 103
  P->>B1: Produce to orders.events-0 with acks=all
  B1-->>P: Ack at offset 880
  Note over B1: Broker 101 crashes
  P->>B1: Produce next batch
  Note over P: No response, the batch stays buffered and is retried
  Note over C: No heartbeat from 101 for broker.session.timeout.ms
  C->>C: Fence 101, set leader 102, ISR 102 103, epoch 8
  B2->>C: Fetch metadata
  C-->>B2: Partition change record
  B3->>C: Fetch metadata
  C-->>B3: Partition change record
  Note over B2: Becomes leader at epoch 8
  B3->>B2: Fetch as follower with epoch 8
  P->>B2: Metadata request
  B2-->>P: Leader of orders.events-0 is 102
  P->>B2: Retry the batch with the same sequence number
  B2-->>P: Ack at offset 881
  Note over B1: Restarts later, truncates to the end of epoch 7, follows 102
A leader crash seen from the producer: a pause, a metadata refresh and a retry.
  1. The pause the producer sees is almost entirely detection: up to broker.session.timeout.ms before the controller acts. The election itself is one metadata record.
  2. Brokers 102 and 103 are not told individually. They fetch the metadata log as usual and act on the partition change record when they replay it.
  3. The producer learns the new leader from any broker's metadata cache, then resends. The same sequence number lets the new leader detect a batch it already has.
  4. Offset 880 was acknowledged with acks=all, so 102 and 103 both had it before 101 failed. Nothing committed is lost, and when 101 returns it removes any uncommitted records it holds beyond what epoch 7 contained on the new leader.

In a real system

A typical order platform: orders.events is keyed by order id, has 12 partitions, replication factor 3 across three availability zones, min.insync.replicas=2, and unclean election disabled. The order service produces with acks=all and idempotence. Brokers are restarted one at a time for an upgrade every few weeks, and occasionally a host fails.

EventWhat the cluster doesWhat the order service sees
Rolling restart of one broker (SIGTERM)Controlled shutdown moves its 4 leaderships to in-sync replicas before it stops. ISR is 2 of 3 until it returns and catches up, which still satisfies min.insync.replicas=2.A few retried requests and a small latency bump. No errors and no duplicates.
One broker's host diesAbout 9 seconds until the controller fences it, then new leaders for its 4 partitions. The other 8 partitions are unaffected.Sends to those 4 partitions stall for some seconds, then complete. Well within the 2 minute delivery timeout.
A second broker dies before the first is backPartitions that had both replicas there are down to 1 in-sync replica. They still have a leader, but acks=all writes are rejected because the ISR is below min.insync.replicas.Not-enough-replicas errors that are retried until the delivery timeout, then failures. Reads still work.
All three replicas of a partition are downThe partition is offline. It gets a leader again when an eligible replica restarts.Sends for keys on that partition time out. The service must keep or reject those orders, not drop them.

The same company's metrics.host-cpu topic makes the opposite choice: replication factor 2, unclean.leader.election.enable=true. A dashboard with a gap of a few seconds is acceptable, and a monitoring topic that is offline during an incident is not. The setting is per topic precisely so that both decisions can live in one cluster.

For the rolling restart to be invisible, three things must hold together: replication factor 3, min.insync.replicas=2 (not 3, or every restart blocks writes), and one broker at a time, waiting for under-replicated partitions to return to zero before the next.

Java and Spring Boot

Application code never takes part in an election. Its job is to be configured so that an election shows up as a delay. Below: a producer set up for that, the topic settings that decide the trade-off, an Admin-based check of leaders and ISR, and the commands an operator uses.

src/main/resources/application.ymlYAML
spring:  kafka:    bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:broker101:9092,broker102:9092,broker103:9092}    producer:      acks: all      key-serializer: org.apache.kafka.common.serialization.StringSerializer      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer      properties:        # Retries during an election must not duplicate or reorder (default: true).        enable.idempotence: true        # Upper bound on send + retries. Must comfortably exceed the time to        # detect a dead broker (broker.session.timeout.ms, 9 s) plus the election.        delivery.timeout.ms: 120000        request.timeout.ms: 30000        retry.backoff.ms: 100        retry.backoff.max.ms: 1000    consumer:      group-id: payment-service      enable-auto-commit: false

Why it is written this way

  • These are the client defaults in current Kafka versions, written out so nobody lowers them by accident. A delivery.timeout.ms of a few seconds turns every broker failure into application errors.
  • Several brokers are listed in bootstrap-servers so the client can still fetch metadata when one of them is the broker that just failed.
  • acks: all only gives durability across a leader change together with the topic's min.insync.replicas and replication factor, set below.
  • Consumers need no special configuration for leader changes: the client refreshes metadata and continues from its position.

Configuration

Broker and topic settings that govern elections, then the client settings that decide how an election is experienced. Defaults are from the Kafka 4.4 configuration reference.

unclean.leader.election.enableDefault false
What it does
Whether a replica outside the ISR may be elected leader as a last resort, at the risk of losing data. Broker-wide default and topic-level override.
When to change it
Leave it false for business data. Enable it per topic only where a gap is cheaper than an outage. Prefer the one-off kafka-leader-election.sh --election-type unclean to leaving it on.
min.insync.replicasDefault 1
What it does
Minimum ISR size for an acks=all write to be accepted. Also what makes eligible leader replicas possible, since the high watermark stops below it.
When to change it
Set 2 with replication factor 3. With 1, a lone leader accepts writes that an election can then lose if that leader fails.
auto.leader.rebalance.enableDefault true
What it does
A background task in the controller moves leadership back to preferred replicas when leadership is unbalanced.
When to change it
Keep it on. Turn it off only if you want to control the timing yourself, and then run preferred elections after every broker restart.
leader.imbalance.check.interval.secondsDefault 300
What it does
How often the controller runs the leader balance check.
When to change it
Explains why a restarted broker leads nothing for some minutes. Run a preferred election by hand if you need balance sooner.
broker.session.timeout.msDefault 9000 (9 seconds)
What it does
Time without a heartbeat after which the controller fences a broker and elects new leaders for its partitions.
When to change it
This is most of the unavailability after a crash. Lowering it shortens failover but makes long garbage collection or network pauses look like failures.
replica.lag.time.max.msDefault 30000 (30 seconds)
What it does
A follower that has not caught up to the leader's log end for this long is removed from the ISR, and so from the set of clean election candidates.
When to change it
Rarely changed. A lower value keeps the ISR strict but shrinks it more often under load.
controlled.shutdown.enableDefault true
What it does
On a graceful stop, the broker has its leaderships moved away before it shuts down.
When to change it
Leave on, and stop brokers with SIGTERM and enough time to finish. It only succeeds if the broker's partitions have other live replicas.
delivery.timeout.ms (producer)Default 120000 (2 minutes)
What it does
Upper bound on the time from send() to success or failure, including all retries.
When to change it
Keep it well above detection plus election time. Spring Boot: spring.kafka.producer.properties.delivery.timeout.ms.
retries (producer)Default 2147483647
What it does
How many times a failed request is retried. In practice delivery.timeout.ms is the limit.
When to change it
Do not set it to 0 to avoid duplicates. Idempotence handles those, and without retries every election becomes an error.
enable.idempotence (producer)Default true
What it does
The broker discards a retried batch it already has, using the producer id and sequence number, and keeps per-partition order with up to 5 in-flight requests.
When to change it
Leave on. It requires acks=all. It covers the producer's retries, not duplicates created by your application or by consumers.
retry.backoff.ms (clients)Default 100
What it does
Initial wait before retrying a failed request, growing up to retry.backoff.max.ms (1000).
When to change it
Leave at the default. It keeps clients from flooding brokers with metadata requests while a leader is being elected.
metadata.max.age.ms (clients)Default 300000 (5 minutes)
What it does
How often a client refreshes metadata when nothing forces it to. A leadership error triggers a refresh at once, regardless of this value.
When to change it
No need to lower it for failover. Lower it only to discover new partitions or topics sooner.

Hands-on exercise

Fail a leader, then fail the whole ISR

See that the controller picks the new leader from the ISR, that a restarted broker returns as a follower, and what the partition looks like when no in-sync replica is left. Then run the same failure with producers and consumers attached in the lab.

Steps · 0 of 6 done

Leader failure and recovery

Simulation · not a real cluster

Stop the broker that leads a partition, watch the controller promote an in-sync replica, then keep stopping brokers until the partition has no eligible leader.

Take it further in the lab: Broker failure and leader election

What can go wrong

Enabling unclean election to end an outage, and leaving it on
Symptom
After an incident, some orders that were acknowledged to the producer are missing from the topic, and a few consumers report positions beyond the end of the log.
Cause
An out-of-sync replica was promoted. Its log became authoritative and the committed records it lacked were truncated on the other replicas when they returned.
Fix
Keep unclean.leader.election.enable=false on business topics. If a partition must be forced, use the one-off kafka-leader-election.sh --election-type unclean for that partition, then reconcile the lost records from the source system.
Expecting acks=1 writes to survive a leader change
Symptom
A small number of records sent just before a broker failure never reach consumers. The producer logged success for them.
Cause
acks=1 is acknowledged by the leader alone. Records not yet replicated exist only there, and a clean election to another replica does not include them.
Fix
Use acks=all with min.insync.replicas=2 and replication factor 3 for anything that must not be lost.
Short producer timeouts
Symptom
Every broker restart produces a burst of send failures (timeout exceptions) in services, although the cluster recovers within seconds.
Cause
delivery.timeout.ms or retries was lowered, so the producer gives up before the controller has detected the failure and elected a leader.
Fix
Restore the defaults (delivery.timeout.ms 120000, unlimited retries) and rely on idempotence to make retries safe. Handle a real send failure explicitly, for example with an outbox.
Killing brokers during a rolling restart
Symptom
Each restart causes several seconds of stalled producers and a slow broker start with log recovery, instead of a barely visible blip.
Cause
The process was stopped with SIGKILL or the stop timeout was too short, so there was no controlled shutdown: the controller had to wait for the session timeout.
Fix
Stop brokers with SIGTERM and give them time. Restart one at a time and wait until under-replicated partitions are back to zero before the next.
min.insync.replicas equal to the replication factor
Symptom
Writes fail with not-enough-replicas errors every time a single broker is down, even for planned maintenance.
Cause
With replication factor 3 and min.insync.replicas=3, losing any replica drops the ISR below the minimum. The partition has a leader but refuses acks=all writes.
Fix
Use min.insync.replicas of replication factor minus one (2 of 3). This is a write-availability problem, not an election problem: check the ISR, not the leader.
Confusing a controller election with partition elections
Symptom
During a controller failover the team expects every partition to change leader and prepares for a traffic interruption that never comes, or the reverse.
Cause
The two are separate. A controller election changes who manages metadata and moves no partition leader. A broker failure moves partition leaders and involves no vote among replicas.
Fix
Diagnose them separately: kafka-metadata-quorum.sh describe --status for the controller, kafka-topics.sh --describe for partition leaders.

How to troubleshoot

Producers time out for some keys only

Check

`bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --unavailable-partitions`, and the metric `kafka.controller:type=KafkaController,name=OfflinePartitionsCount`
How to read it
Listed partitions have no leader: every replica that could lead is down. The Isr column shows which broker must come back. A count above 0 is an outage for the keys that hash to those partitions.
Writes fail with not-enough-replicas errors but reads work

Check

`bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-min-isr-partitions`, and the metric `kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount`
How to read it
The partition has a leader, but its ISR is smaller than min.insync.replicas. Find out why followers left the ISR (broker down, or lagging beyond replica.lag.time.max.ms). Electing a different leader will not help.
One broker carries much more traffic than the others after a restart elsewhere

Check

`kafka.controller:type=KafkaController,name=PreferredReplicaImbalanceCount`, per-broker `kafka.server:type=ReplicaManager,name=LeaderCount`, and the `Leader` column against the first entry of `Replicas` in `kafka-topics.sh --describe`
How to read it
Leadership has not moved back to preferred replicas yet. The automatic check runs every 5 minutes, or run bin/kafka-leader-election.sh --bootstrap-server localhost:9092 --election-type preferred --all-topic-partitions.
You need to know whether elections are happening and whether any was unclean

Check

Metrics `kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs` and `kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec`
How to read it
The election rate is non-zero when brokers fail or restart and should otherwise be flat. The unclean rate must be 0: any other value means data was lost on some partition.
Client logs fill with NOT_LEADER_OR_FOLLOWER or LEADER_NOT_AVAILABLE

Check

Whether the messages stop within seconds, and `bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --replication` if they do not
How to read it
A short burst is the normal sign of a leadership change: the client is refreshing metadata and retrying. If it continues, some broker is serving stale metadata or the partition really has no leader.
Under-replicated partitions stay above zero after a broker returned

Check

`bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions`, and `kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions` with `IsrExpandsPerSec`
How to read it
The returned broker is still catching up as a follower and is not yet an election candidate. Do not restart the next broker until this reaches zero.

Key takeaways

  • The active controller elects partition leaders, from the ISR. Replicas do not vote. Controller election is a separate, majority-vote mechanism for the metadata quorum.
  • Any ISR member has every committed record, so a clean election loses nothing that was acknowledged with acks=all.
  • Leadership moves on broker failure (after the session timeout), controlled shutdown (almost immediately), preferred election and reassignment.
  • Each new leader gets a higher leader epoch. Epochs fence stale leaders and tell returning replicas exactly what to truncate, so replicas never keep diverging logs.
  • With no eligible replica the partition is offline by default. Unclean election trades committed data for availability and should be a per-topic, deliberate choice.
  • Clients ride through an election by refreshing metadata and retrying within delivery.timeout.ms. Idempotence makes those retries safe.
  • Records acknowledged with acks=1 can be lost in a perfectly clean election.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

Partition payments.captured-4 has replicas 101, 102, 103. Broker 103 was removed from the ISR 20 minutes ago because its disk was slow. Then a rack power failure took down 101 and 102 together. Broker 103 is running. The partition shows no leader, producers for some payment keys are timing out, and a manager asks you to run an unclean election now. Power is expected back in about 15 minutes.

What do you do?

Scenario 2 of 3

During a rolling upgrade, each broker restart causes the checkout service to log about 200 TimeoutException send failures, and each restart is followed by a 10 minute log recovery on the broker. The service sets delivery.timeout.ms=5000 and retries=1 'to fail fast'. The deployment tool stops brokers with a 5 second grace period and then kills them.

Which change removes the failures?

Scenario 3 of 3

Broker 102 had a 40 second stop-the-world pause. When it resumed, it still believed it was the leader of orders.events-7 with epoch 12. Meanwhile the controller had fenced it and made 103 the leader with epoch 13. A producer with stale metadata sends a batch to 102 with acks=all right after the pause ends.

What happens to that batch?

References

References: Apache Kafka documentation: Design, Replication (ISR, unclean leader election, replica management) ↗ · Apache Kafka documentation: Basic Kafka operations (graceful shutdown, balancing leadership) ↗ · Apache Kafka documentation: Eligible Leader Replicas ↗ · Apache Kafka documentation: Broker configuration reference ↗ · Apache Kafka documentation: Producer configuration reference ↗ · Apache Kafka documentation: Monitoring (controller and replica manager metrics) ↗ · KIP-101: Alter Replication Protocol to use Leader Epoch rather than High Watermark for Truncation ↗ · KIP-320: Allow fetchers to detect and handle log truncation ↗ · KIP-460: Admin Leader Election RPC ↗ · KIP-631: The Quorum-based Kafka Controller (broker heartbeats, fencing, controlled shutdown) ↗ · KIP-966: Eligible Leader Replicas ↗ · Apache Kafka Javadoc: Admin (describeTopics, electLeaders) ↗ · Report a problem