Skip to content

Path 10 · lesson 2 of 6

Advanced30 minApache Kafka 4.x, KRaft only (checked against the 4.4 Operations, Design and configuration pages, and KIP-966) · Java 21 · Spring Boot 3.5 with Spring for Apache Kafka 3.3 and kafka-clients 3.9

Broker failures and incident response

What a cluster does when a broker crashes, returns, fills its disk or merely slows down, and a calm sequence for handling the incident, including reassignment.

What you will learn

  • Trace a broker crash step by step: detection, fencing, leader election, client recovery
  • Describe what a returning broker does before it is useful again, and why leadership does not move back at once
  • Recognise a full disk, a slow broker and a lost controller quorum from their different symptoms
  • Run an incident in a fixed order: stabilise, assess, decide, communicate, review
  • Decide deliberately whether to run an unclean leader election and state what it costs
  • Move replicas off a failed broker with kafka-reassign-partitions.sh and a throttle

Before you start

The concept

A broker failure is the event Kafka's replication was designed for. With replication factor 3 and min.insync.replicas=2, losing one broker should cost clients a few seconds of retries and nothing else. An incident is what you get when something sits outside that design: a second failure before the first is repaired, a broker that is sick but not dead, a disk with no space left, or a person making a quick decision that trades data for availability without realising it.

So this lesson has two halves. The first is mechanical: exactly what the cluster does by itself, so you know what to leave alone. The second is procedural: what you do, in what order, when the automatic behaviour is not enough.

FailureWhat Kafka does by itselfWhat clients seeWhat needs a person
Broker crashController fences it after the session timeout and elects new leaders from the ISR.A few seconds of retriable errors, then normal service.Bring the broker back, or replace it and move its replicas.
Broker returnsIt re-registers, truncates and catches up, rejoins ISRs, and later gets its preferred leaderships back.Nothing, if catch-up is not starving live traffic.Wait for under-replicated partitions to reach 0 before other work.
Full diskThe log directory goes offline. A broker with no usable directory shuts down.As for a crash, but the broker cannot simply be restarted.Free or add space. Kafka cannot do this for itself.
Slow brokerVery little. It still heartbeats, so it is not fenced and keeps its leaderships.High latency and timeouts on the partitions it leads, for as long as it stays up.Notice it, then stop it on purpose so the crash path takes over.
Controller quorum lostNothing. Brokers serve with the metadata they have.Nothing at first. Then any further failure cannot be handled.Restore a majority of controllers before anything else.
All in-sync replicas downWith the default settings, the partition stays offline and waits.Timeouts on that partition.Recover a replica that has the data, or choose an unclean election knowingly.

A crash, a return and a second failure

Simulation · not a real cluster

Stop the broker that leads a partition and read the event log line by line. Restart it and notice that it comes back as a follower. Then stop two brokers and see what changes for acks=all.

Why it exists

Why does the automatic handling need to be understood in this much detail? Because during an incident the most damaging actions are the well-meant ones. Restarting a second broker while the first is still catching up. Starting a large reassignment without a throttle on a cluster that is already short of capacity. Lowering min.insync.replicas to make producer errors stop. Enabling unclean leader election to get a partition back, and discovering later that acknowledged orders are gone. Each of these looks like a fix and each makes a recoverable situation worse.

A fixed order of work protects against this. It forces two questions before any change: what is the cluster already doing about this by itself, and does the action I am about to take reduce the number of in-sync replicas anywhere?

Availability first

  • Get the partition serving again as fast as possible
  • Accept that some acknowledged records may be lost
  • Reasonable for metrics, clickstream, logs that are sampled anyway
  • Mechanism: an unclean leader election

Consistency first

  • Keep the partition offline until a replica with all committed data returns
  • Accept an outage that lasts as long as the repair
  • Required for orders, payments, anything reconciled against another system
  • Mechanism: the default, unclean.leader.election.enable=false

Neither side is correct in general. What matters is that the choice is made per topic, in advance where possible, and by someone who can say what the lost records were worth. The Kafka design documentation frames it the same way: a trade-off between availability and consistency that exists in every replicated system.

How it works inside

A broker crash, step by step

  1. Heartbeats stop. Each broker sends a heartbeat to the active controller every broker.heartbeat.interval.ms (2 seconds). A crashed broker sends none.
  2. The session expires. After broker.session.timeout.ms (9 seconds) without a heartbeat, the controller fences the broker: it is still registered, but it may not lead partitions and is not offered to clients.
  3. The controller rewrites partition state. For every partition with a replica on that broker it removes the broker from the ISR. For every partition the broker led, it elects a new leader, choosing a replica from the remaining ISR, and increments the leader epoch. These changes are records in the metadata log.
  4. Brokers apply the change. The new leaders start accepting produce and fetch requests. Followers begin fetching from them, truncating first if their log is ahead of what the new leader has.
  5. Clients find the new leader. A producer or consumer still talking to the dead broker gets a connection error or a request timeout; one talking to a broker that is no longer leader gets NotLeaderOrFollowerException. Both are retriable and both make the client refresh its metadata and retry against the new leader.

From the client's side the gap is the detection time plus one metadata refresh, on the order of ten seconds with default settings. A producer with idempotence and the default delivery.timeout.ms of 2 minutes rides through it: records wait in its buffer and are sent, in order and without duplicates, to the new leader. A producer with a short delivery timeout or retries=0 reports failures instead. Consumers pause on the affected partitions and resume from their position, which is why a single broker failure should never appear as lost or duplicated data by itself.

A controlled shutdown (the broker process is stopped normally) skips the waiting. The broker tells the controller it is leaving, the controller moves its leaderships first, and only then does the process exit. controlled.shutdown.enable is true by default. The documentation notes two benefits: leadership moves within milliseconds per partition, and the logs are flushed so no recovery is needed at restart. This is why killing a broker with kill -9 is never an acceptable way to restart one.

When the broker returns

  1. Log recovery. After an unclean stop the broker validates the tail of each log before using it. On a broker with many partitions this takes time; num.recovery.threads.per.data.dir (default 2) sets the parallelism and the remainingLogsToRecover metric shows progress. After a controlled shutdown this step is skipped.
  2. Registration and metadata catch-up. The broker registers with the controller and replays the metadata log to learn what changed while it was away. It stays fenced until it has caught up.
  3. Follower for everything. Once unfenced it is a follower for all its partitions, including those it used to lead. It truncates any records it holds beyond the new leader's history for that epoch, then fetches what it missed.
  4. Rejoining the ISR. When a replica has caught up to the leader's log end, the leader proposes adding it to the ISR. Under-replicated partitions fall back towards 0.
  5. Preferred leader election. The first broker in a partition's replica list is its preferred leader. With auto.leader.rebalance.enable=true (the default) the controller checks every leader.imbalance.check.interval.seconds (300) and hands leadership back to preferred replicas that are in the ISR. Until then the returned broker serves no client traffic as a leader.

When no in-sync replica is left

If every replica in a partition's ISR is on a stopped broker, the partition is offline. A replica that is running but was not in the ISR may be missing committed records, so by default it is not allowed to lead. From Kafka 4.0 the controller can also track eligible leader replicas (ELR, KIP-966): replicas that left the ISR while the partition was already below min.insync.replicas. Because the high watermark cannot advance in that state, such a replica is known to hold everything that was committed and is safe to elect. ELR is enabled by default on clusters created with 4.1 or later, and can be enabled on upgraded clusters. The election order is: a member of the ISR; otherwise an unfenced member of the ELR; otherwise wait.

An unclean leader election breaks the wait by electing a replica that is neither. Its log becomes the truth for the partition. Every record it lacks is gone, including records that producers were told were committed, and when the old leader returns it truncates its log to match. Consumers that had already read those records now hold offsets beyond the end of the new log. unclean.leader.election.enable is false by default and can be set per topic. Setting it dynamically does not act at once: the controller applies it on a periodic check, so for an immediate effect on chosen partitions the documentation points to kafka-leader-election.sh --election-type unclean.

A full disk

When a write to a log directory fails, the broker marks that directory offline (OfflineLogDirectoryCount), stops serving the replicas in it and reports this to the controller, which elects other leaders. A broker with several directories in log.dirs keeps running on the remaining ones. A broker whose only directory, or every directory, has failed shuts itself down. What makes a full disk harder than a crash is that restarting does not help: there is still no space, and recovery itself needs to write.

Space is released only by deleting whole log segments, which happens when retention says so. The safe ways out are to add capacity to the volume, or, while the brokers are still running, to shorten retention on the largest topics with kafka-configs.sh and let the broker delete old segments itself. Deleting files by hand from a partition directory is how a full disk turns into data loss. The Break card Too full to empty shows the same trap in a different queueing system: a store so full that the operation that would drain it cannot run.

A slow broker

Typical causes are a degrading disk, a noisy neighbour, a network interface dropping packets, or long garbage collection pauses. The signs are indirect. Partitions led by the slow broker show high produce latency, with the time in LocalTimeMs on that broker. Partitions it follows show high RemoteTimeMs on their leaders, and it keeps dropping out of ISRs and rejoining. Producers see timeouts for some partitions and not others. Consumer lag grows on the partitions it leads. If you can see that one broker is the common factor and it is not recovering, stop it with a controlled shutdown. The cluster then handles a problem it knows how to handle.

Losing the controller quorum

If fewer than a majority of controllers are running, there is no active controller. Brokers keep their last metadata, existing leaders keep accepting writes, and consumers keep reading. But nothing that needs a metadata change can happen: no leader election, no ISR shrink or expansion, no topic creation, no broker registration. A cluster in this state looks healthy to its clients and has no ability to respond to the next failure. Restoring the quorum comes before every other task. Do not restart brokers while it is down, because a restarted broker cannot register and will not come back.

Controller quorum

Simulation, not a real cluster

Fail a controller and watch a new one elected by majority, then see what happens when a majority is not available.

Step 1 · The pattern

One controller runs the cluster

Brokers store the data, and one active controller decides which broker leads each partition. Every broker follows the same controller, so every client gets the same map of the cluster.

  • write
  • confirmed
Open the full animated lesson

An incident-response sequence

  1. Stabilise. Stop changing things. Pause deployments, restarts and reassignments that are in flight. Confirm there is an active controller. If one broker is identified as slow or flapping, stop it cleanly. Do not start anything that removes another replica.
  2. Assess, worst first. List offline partitions, then under-min-ISR partitions, then under-replicated ones, and work out from the Replicas and Isr columns which brokers are missing. This tells you who is affected (which topics, so which services) and what would fix it (which broker has to return).
  3. Decide about data deliberately. If a replica with the committed data can be brought back in acceptable time, wait for it. An unclean election or a lowered min.insync.replicas is a decision for the owner of the data, taken per topic, and recorded with the time and the reason.
  4. Communicate. Tell the teams that produce to and consume from the affected topics what is refused, what is delayed and what, if anything, may be lost. Producers need to know whether their buffered or failed sends will need replaying; consumers need to know whether to expect a backlog.
  5. Recover fully. Bring replicas back, replace what cannot be repaired with a throttled reassignment, wait for under-replicated partitions to reach 0, and rebalance leadership.
  6. Review. Write down the timeline, which signal should have fired earlier, and which manual step should become a check in tooling.

Partition reassignment

Kafka does not move replicas by itself. If a broker is gone for good, its partitions stay under-replicated until you change their replica lists. kafka-reassign-partitions.sh does this in three mutually exclusive modes: --generate proposes a plan for a list of topics and target brokers, --execute starts a plan given as JSON, and --verify reports whether each partition has finished. --list shows reassignments in progress and --cancel stops one.

Mechanically, the controller adds the new replica to the partition as a follower. It copies the whole partition from the leader. When it has caught up and joined the ISR, the replica being removed is dropped and its data deleted. The copy is the expensive part: it reads old segments from the leader's disk and uses network on both sides. --throttle (bytes per second) caps that traffic so it cannot starve producers and consumers. Two rules from the documentation matter. The throttle must be higher than the rate at which the partition is being written, or the new replica never catches up. And --verify must be run after completion, because it is what removes the throttle. A throttle left in place slows ordinary replication later.

Architecture

Drawing diagram…

A state diagram for one partition. Healthy becomes under-replicated when one replica is lost and returns when it rejoins the ISR. A second loss makes it under min ISR, where acks=all writes are refused but reads work. Losing the last in-sync replica makes it offline. It leaves offline either cleanly, when an in-sync or eligible replica returns, or through an unclean leader election, which leads to a data loss state before other replicas truncate and the partition is under-replicated again.

stateDiagram-v2
  [*] --> Healthy
  Healthy --> UnderReplicated: one replica lost or lagging
  UnderReplicated --> Healthy: replica catches up and rejoins the ISR
  UnderReplicated --> UnderMinIsr: second replica lost
  UnderMinIsr --> UnderReplicated: one replica rejoins the ISR
  UnderMinIsr --> Offline: last in-sync replica lost
  Offline --> UnderMinIsr: an in-sync or eligible replica returns
  Offline --> DataLoss: unclean leader election
  DataLoss --> UnderReplicated: other replicas truncate to the new leader
  note right of UnderReplicated
    Reads and writes work
    Margin is reduced
  end note
  note right of UnderMinIsr
    acks=all writes refused
    Reads still work
  end note
  note right of Offline
    No leader
    Nothing works on this partition
  end note
The states of one partition with replication factor 3 and min.insync.replicas=2, and what moves it between them.
  • Each step down the left side is one more replica lost. Kafka handles the first without help, refuses durable writes at the second, and stops at the third.
  • There are two ways out of the offline state and they are very different. The clean one waits for a replica that has the data. The other accepts whatever the chosen replica has.
  • The diagram is per partition. During one incident different partitions are in different states, which is why the first assessment step is to list them.
  • Most actions taken in a hurry (restarting another broker, an unthrottled reassignment) push partitions down this diagram, not up.

Message flow

Drawing diagram…

A sequence diagram. A producer writes to the leader on broker 1 and is acknowledged. Broker 1 crashes and the next batch gets no response and stays buffered. After nine seconds without a heartbeat the controller fences broker 1, shrinks the ISR, makes broker 2 the leader and increments the epoch. The producer refreshes metadata, learns the new leader and retries the batch with the same sequence numbers, which is acknowledged. Broker 1 restarts, registers, fetches as a follower from broker 2, is added back to the ISR, and later a preferred election returns leadership to it.

sequenceDiagram
  participant P as Order Service producer
  participant B1 as Broker 1 leader
  participant C as Active controller
  participant B2 as Broker 2 follower
  P->>B1: Produce to orders.events-0 with acks=all
  B1-->>P: Acknowledged
  Note over B1: Broker 1 crashes
  P->>B1: Produce next batch
  Note over P: No response, batch stays buffered
  C->>C: No heartbeat from broker 1 for 9 seconds
  C->>C: Fence broker 1, ISR becomes 2 and 3, leader is 2, epoch plus one
  C-->>B2: Metadata record, you lead orders.events-0
  P->>B2: Metadata request after timeout
  B2-->>P: Leader of orders.events-0 is broker 2
  P->>B2: Retry the buffered batch with the same sequence numbers
  B2-->>P: Acknowledged
  Note over B1: Broker 1 restarts
  B1->>C: Register and catch up on metadata
  B1->>B2: Fetch as follower, truncate then copy
  B2->>C: Propose ISR 2 and 3 and 1
  C->>C: Later, preferred election returns leadership to broker 1
One broker crash from the producer's point of view, and the broker's return.
  1. The batch sent to the dead broker is not lost and not duplicated. It waits in the producer and is retried with the same sequence numbers, so the idempotent producer and the new leader agree on what has been written.
  2. Between the crash and the fencing, about nine seconds with defaults, the partition has a leader on paper that is not answering. This window is the main part of what clients experience.
  3. The returning broker does not resume where it stopped. It learns from the metadata that it is now a follower, and brings its log in line with the new leader before copying.
  4. Leadership returns to broker 1 only after it is back in the ISR and the periodic preferred election runs. Until then broker 2 carries extra leader load.

In a real system

A typical incident at a payments company. The cluster has three brokers, one per availability zone, and three dedicated controllers. Topic payments.events has 12 partitions, replication factor 3 and min.insync.replicas=2, keyed by payment id. The Payment Service produces with acks=all; the Ledger Service and the Notification Service consume it in separate groups.

TimeEventState of payments.eventsDecision
02:10Broker 3's data volume starts returning I/O errors. Its log directory goes offline and the broker shuts down.Under-replicated: ISR is 1,2 on all 12 partitions. Writes continue.A ticket is opened for the morning. Reasonable: no client is affected.
08:45A new volume is attached and broker 3 starts with an empty disk. It begins copying all 12 partitions from the leaders.Still under-replicated. Leaders on 1 and 2 are now also serving a full copy.Nobody limits or watches the catch-up traffic.
09:20Morning traffic peaks. Broker 2, serving clients and the copy, has request handler idle below 10%. An engineer restarts it, reading the alert as a hung broker.Under min ISR: ISR is 1 only. Every acks=all write is refused.The wrong action: it removed an in-sync replica while another was missing.
09:21Payment Service returns errors to checkout. Page fires on under-min-ISR partitions.Under min ISR.Incident declared. One person takes command, one handles communication.
09:24Someone proposes setting min.insync.replicas=1 on the topic so that writes resume.Under min ISR.Declined. Broker 1 would be the only copy of every new payment event. Broker 2 is restarting and is minutes away.
09:31Broker 2 finishes its controlled restart and catches up on nine minutes of nothing: no writes were accepted.Under-replicated: ISR is 1,2. Writes resume.Checkout recovers. Status update sent with the cause as currently understood.
11:50Broker 3 completes its copy and rejoins every ISR.Healthy.Incident closed only now.
Illustrative timeline of a typical incident. The outage was ten minutes; the period of reduced safety was nearly ten hours.

The review found three things worth changing, none of which was the disk. A broker that returns empty should have its catch-up limited so that it cannot overload the leaders it copies from. An overloaded broker and a hung broker look alike on a single graph, so the runbook now says to check under-replicated partitions before restarting anything, and to never restart a broker while that number is not 0 unless the broker is the cause. And the question about lowering min.insync.replicas now has a written answer per topic, agreed with the business owners in advance, so it is not argued at 09:24.

Java and Spring Boot

The commands for assessing an incident, the reassignment that replaces a lost broker, and the producer settings that decide whether an application rides through a leader change or reports errors. Sample output is illustrative, not captured from a real cluster.

incident/01-assess.shShell
BS=broker1.example.com:9092# 1. Is there an active controller? Without one nothing below can change.bin/kafka-metadata-quorum.sh --bootstrap-server $BS describe --status# Illustrative output:#   LeaderId:               3001#   LeaderEpoch:            14#   HighWatermark:          982113#   MaxFollowerLag:         0#   CurrentVoters:          [{"id": 3000, ...}, {"id": 3001, ...}, {"id": 3002, ...}]# 2. Worst first: partitions with no leaderbin/kafka-topics.sh --bootstrap-server $BS --describe --unavailable-partitions# 3. Partitions refusing acks=all writesbin/kafka-topics.sh --bootstrap-server $BS --describe --under-min-isr-partitions# Illustrative output:#   Topic: payments.events  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1#   Topic: payments.events  Partition: 1  Leader: 1  Replicas: 2,3,1  Isr: 1# 4. Everything with reduced marginbin/kafka-topics.sh --bootstrap-server $BS --describe --under-replicated-partitions# 5. Is a reassignment already running? (It would explain extra load.)bin/kafka-reassign-partitions.sh --bootstrap-server $BS --list

Why it is written this way

  • The order is by severity. An offline partition and a missing controller change what you do next; under-replication alone usually means wait and watch.
  • In each output line, the broker ids present in Replicas and absent from Isr are the ones to investigate. Here brokers 2 and 3 are missing everywhere.
  • If brokers are unreachable, the quorum tool also accepts --bootstrap-controller with a controller address.
  • On 4.x the describe output may also print Elr and LastKnownElr fields. A replica listed under Elr can be elected without data loss.

Configuration

The settings that decide how quickly a failure is detected, what the cluster does about it, and what a returning broker does. Defaults are from the Apache Kafka 4.4 configuration reference.

broker.session.timeout.msDefault 9000 (9 seconds)
What it does
How long the controller waits without a heartbeat before fencing a broker and moving its leaderships.
When to change it
The main part of client-visible failover time after a crash. Lowering it gives faster failover and more false fencing during GC pauses or network blips. Change it only with evidence.
broker.heartbeat.interval.msDefault 2000 (2 seconds)
What it does
How often a broker sends a heartbeat to the active controller.
When to change it
Keep it well below the session timeout so that a few lost heartbeats do not fence a healthy broker.
replica.lag.time.max.msDefault 30000 (30 seconds)
What it does
A follower that has not caught up to the leader's log end for this long is removed from the ISR.
When to change it
This is how a slow follower is handled, as opposed to a dead broker. The producer's request.timeout.ms is documented as needing to be larger than this value.
unclean.leader.election.enableDefault false
What it does
Whether a replica outside the ISR may be elected leader as a last resort, at the risk of losing committed records. Broker default and topic-level setting.
When to change it
Leave false for data that must not be lost. Consider true per topic for data where availability matters more. A dynamic change takes effect on a periodic controller check, not at once.
min.insync.replicasDefault 1
What it does
The smallest ISR size at which acks=all writes are accepted.
When to change it
2 with replication factor 3 tolerates one failure with no loss of durability. Lowering it during an incident restores writes by allowing a single copy: a decision for the data owner, not a reflex.
controlled.shutdown.enableDefault true
What it does
On a normal stop, the broker asks the controller to move its leaderships away before the process exits.
When to change it
Leave on. Make sure your process manager gives the broker enough time to finish before it sends a hard kill.
auto.leader.rebalance.enableDefault true
What it does
The controller periodically moves leadership back to each partition's preferred replica when that replica is in the ISR.
When to change it
Leave on unless you run preferred elections yourself with kafka-leader-election.sh --election-type preferred. Without either, leaders pile up on the brokers that did not restart.
leader.imbalance.check.interval.secondsDefault 300
What it does
How often the controller runs the check behind auto.leader.rebalance.enable.
When to change it
Explains why a returned broker leads nothing for up to five minutes after it is back in sync. Rarely worth changing.
num.recovery.threads.per.data.dirDefault 2
What it does
Threads per log directory used for log recovery at startup and for flushing at shutdown. The default was 1 before Kafka 4.0.
When to change it
Raise it on brokers with many partitions and fast disks to shorten restart after a crash.
delivery.timeout.ms (producer)Default 120000 (2 minutes)
What it does
Upper bound on the time from send() returning to success or failure being reported, including retries.
When to change it
Must comfortably exceed failover time. If your service cannot wait that long, lower it and handle the failure explicitly; do not disable retries.

Hands-on exercise

Troubleshoot: three brokers are up, and the topic is still offline

Work from an incident channel's evidence to the cause of an outage that survives restarting brokers, decide what to do about it, then run the same failure in the broker-failure lab.

Steps · 0 of 7 done

Reproduce the outage

Simulation · not a real cluster

Three brokers, orders.events with 3 partitions, replication factor 3, min.insync.replicas 2. Follow the incident timeline from the exercise steps.

Take it further in the lab: Broker failure and leader election

What can go wrong

Restarting brokers to see if it helps
Symptom
An incident with producer errors on some partitions becomes one with errors on all of them shortly after an engineer restarts a broker that looked unhealthy.
Cause
The cluster was already under-replicated. The restarted broker held the second in-sync replica of many partitions, so stopping it took them below min.insync.replicas.
Fix
Check under-replicated and under-min-ISR partitions before stopping any broker. Stop a broker during an incident only when it is identified as the cause.
Unclean election as a reflex
Symptom
After an outage, downstream services report missing events, or consumers fail with offset-out-of-range errors and reset to the earliest or latest offset.
Cause
A replica outside the ISR was made leader to end the outage. Records it had not replicated were discarded, including acknowledged ones.
Fix
Keep unclean.leader.election.enable=false for durable topics. Decide per topic in advance who may authorise an unclean election, and check Isr and Elr for a clean candidate first.
Lowering min.insync.replicas to stop the errors
Symptom
Producer errors stop during the incident. Later, a second broker failure loses the events written in between.
Cause
With min.insync.replicas=1, acks=all is satisfied by the leader alone. Every write during that period existed on one disk.
Fix
Treat it as a deliberate, time-boxed choice made by the data owner, recorded in the incident log, and reverted as soon as a second replica is in sync.
Unthrottled or forgotten reassignment
Symptom
Produce latency and consumer lag rise across the cluster during a reassignment; or, weeks later, followers lag for no obvious reason.
Cause
Copying whole partitions competes with client traffic for disk and network. In the second case a throttle was set and --verify was never run, so ordinary replication is still rate-limited.
Fix
Always pass --throttle, sized above the topic's write rate and below the spare capacity. Run --verify until complete, and check kafka-configs.sh --describe --entity-type brokers for leftover throttle settings.
Declaring recovery when the process is up
Symptom
The incident is closed. An hour later a routine deployment restarts another broker and writes are refused again.
Cause
The returned broker was still catching up and was not in any ISR. The cluster had no more margin than during the incident.
Fix
Close on under-replicated partitions at 0 and leadership rebalanced, not on process status. Block maintenance automatically while under-replicated partitions exist.
Freeing disk space by deleting files
Symptom
After a full-disk incident a broker fails to start, or partitions report missing or corrupt segments.
Cause
Segment or index files were removed by hand from a partition directory while the broker's own view of the log still included them.
Fix
Add capacity, or reduce retention.ms / retention.bytes on the largest topics with kafka-configs.sh and let the broker delete segments. Restore the original retention afterwards.

How to troubleshoot

Producers get NotLeaderOrFollowerException or timeouts for a few seconds, then recover

Check

`IsrShrinksPerSec` on the surviving brokers and `FencedBrokerCount` on the controller at that time; the controller log for the broker being fenced; `kafka-topics.sh --bootstrap-server … --describe --topic orders.events`
How to read it
A normal leader change after a broker stop. If it recovers within the producer's delivery timeout and no send failed, nothing is wrong with the clients. Find out why the broker stopped.
Producers fail with NotEnoughReplicasException

Check

`kafka-topics.sh --bootstrap-server … --describe --under-min-isr-partitions`
How to read it
Two replicas of those partitions are missing or lagging. Compare Replicas with Isr to find which brokers. Getting one of them back in sync restores writes; consumers were never blocked.
A partition has no leader although some of its replicas are running

Check

`kafka-topics.sh --bootstrap-server … --describe --unavailable-partitions`, reading the `Isr` and `Elr` columns
How to read it
The running replicas are not in the ISR (or ELR), so electing them is not safe. The brokers listed in Isr are the ones that hold the committed data: bring one of those back.
A restarted broker has been up for a long time and partitions are still under-replicated

Check

`remainingLogsToRecover` on that broker; `kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica`; `ReplicationBytesInPerSec`; leftover throttles with `kafka-configs.sh --bootstrap-server … --describe --entity-type brokers`
How to read it
Non-zero logs to recover: it is still validating its logs after an unclean stop. Lag falling steadily: it is copying and needs time. Lag flat or rising: it cannot fetch faster than the leaders are written to, often because of a forgotten replication throttle or a slow disk.
Latency and timeouts on some partitions only, all brokers running

Check

`kafka.network:type=RequestMetrics,name=LocalTimeMs,request=Produce` per broker; which broker leads the affected partitions; `IsrShrinksPerSec`; host disk latency and network errors
How to read it
If one broker leads all the affected partitions and its local time stands out, it is a slow broker. It will not be fenced while it heartbeats. A controlled shutdown moves its leaderships to healthy replicas.
A broker stopped and will not stay up after restart

Check

Its server log for a log directory failure or `Shutdown broker because all log dirs in … have failed`; free space on the volumes in `log.dirs`; `OfflineLogDirectoryCount` before it stopped
How to read it
A failed or full log directory. Restarting cannot fix it. Repair or extend the volume first; if the disk is lost, start the broker empty and let it copy, with catch-up traffic limited.
Topic creation hangs, a stopped broker's partitions never get new leaders, a restarted broker never becomes active

Check

`kafka-metadata-quorum.sh --bootstrap-controller controller1.example.com:9093 describe --status`; `ActiveControllerCount` summed over the controllers
How to read it
No active controller: the quorum lacks a majority. Existing leaders still serve, which hides the problem. Restore the controllers before touching any broker.

Key takeaways

  • One broker failure with replication factor 3 and min ISR 2 is handled by the cluster: about ten seconds of retries with default timeouts, no loss, no duplicates from an idempotent producer.
  • A returned broker is a follower that must catch up. Recovery is complete when under-replicated partitions are 0 and leadership is rebalanced.
  • A slow broker is not fenced and hurts clients for as long as it runs. Stopping it deliberately is a valid fix.
  • Without a controller majority the cluster serves but cannot react. Restore the quorum first.
  • An unclean leader election and a lowered min.insync.replicas both trade acknowledged data for availability. Make that choice per topic, on purpose, and write it down.
  • Kafka does not move replicas by itself. Use kafka-reassign-partitions.sh with a throttle, and run --verify to finish and remove it.
  • Incident order: stabilise, list offline and under-min-ISR partitions, decide about data deliberately, communicate, review.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

Topic orders.events: replication factor 3, min.insync.replicas=2. Broker 5 has been down for an hour for a hardware repair, so 40 partitions show ISR size 2. Broker 2's request handler idle ratio drops to 0.05 at peak and produce latency on its partitions is 800 ms. A colleague wants to restart broker 2 to clear it.

What happens to the partitions that have replicas on both broker 2 and broker 5 if broker 2 is restarted now?

Scenario 2 of 3

Partition clicks.raw-7 and partition payments.events-3 are both offline after a double failure. For each, describe shows Replicas 4,5,6 and Isr 6, with broker 6 needing a disk replacement that takes about three hours. Brokers 4 and 5 are running. clicks.raw feeds a dashboard with sampled analytics; payments.events feeds the ledger.

What is the sound decision?

Scenario 3 of 3

Broker 3 died permanently. You start a reassignment moving its 200 replicas to a new broker 7 with --throttle 10000000 (10 MB/s). After two hours --verify still reports most partitions in progress, and the replica lag for the busiest topic on broker 7 is growing. That topic receives about 25 MB/s from producers across its partitions.

What explains this, and what do you do?

References

References: Apache Kafka documentation: Basic Kafka operations (graceful shutdown, balancing leadership, reassignment, throttling) ↗ · Apache Kafka documentation: Design (replication, unclean leader election, availability and durability) ↗ · Apache Kafka documentation: Eligible Leader Replicas ↗ · Apache Kafka documentation: Broker configuration reference ↗ · Apache Kafka documentation: Producer configuration reference ↗ · Apache Kafka documentation: Operations, KRaft (quorum tool) ↗ · Apache Kafka documentation: Monitoring ↗ · KIP-966: Eligible Leader Replicas ↗ · KIP-455: Create an Administrative API for Replica Reassignment ↗ · Report a problem