Path 10 · lesson 2 of 6
Broker failures and incident response
What a cluster does when a broker crashes, returns, fills its disk or merely slows down, and a calm sequence for handling the incident, including reassignment.
What you will learn
- Trace a broker crash step by step: detection, fencing, leader election, client recovery
- Describe what a returning broker does before it is useful again, and why leadership does not move back at once
- Recognise a full disk, a slow broker and a lost controller quorum from their different symptoms
- Run an incident in a fixed order: stabilise, assess, decide, communicate, review
- Decide deliberately whether to run an unclean leader election and state what it costs
- Move replicas off a failed broker with
kafka-reassign-partitions.shand a throttle
Before you start
The concept
A broker failure is the event Kafka's replication was designed for. With replication factor 3 and min.insync.replicas=2, losing one broker should cost clients a few seconds of retries and nothing else. An incident is what you get when something sits outside that design: a second failure before the first is repaired, a broker that is sick but not dead, a disk with no space left, or a person making a quick decision that trades data for availability without realising it.
So this lesson has two halves. The first is mechanical: exactly what the cluster does by itself, so you know what to leave alone. The second is procedural: what you do, in what order, when the automatic behaviour is not enough.
| Failure | What Kafka does by itself | What clients see | What needs a person |
|---|---|---|---|
| Broker crash | Controller fences it after the session timeout and elects new leaders from the ISR. | A few seconds of retriable errors, then normal service. | Bring the broker back, or replace it and move its replicas. |
| Broker returns | It re-registers, truncates and catches up, rejoins ISRs, and later gets its preferred leaderships back. | Nothing, if catch-up is not starving live traffic. | Wait for under-replicated partitions to reach 0 before other work. |
| Full disk | The log directory goes offline. A broker with no usable directory shuts down. | As for a crash, but the broker cannot simply be restarted. | Free or add space. Kafka cannot do this for itself. |
| Slow broker | Very little. It still heartbeats, so it is not fenced and keeps its leaderships. | High latency and timeouts on the partitions it leads, for as long as it stays up. | Notice it, then stop it on purpose so the crash path takes over. |
| Controller quorum lost | Nothing. Brokers serve with the metadata they have. | Nothing at first. Then any further failure cannot be handled. | Restore a majority of controllers before anything else. |
| All in-sync replicas down | With the default settings, the partition stays offline and waits. | Timeouts on that partition. | Recover a replica that has the data, or choose an unclean election knowingly. |
A crash, a return and a second failure
Simulation · not a real clusterStop the broker that leads a partition and read the event log line by line. Restart it and notice that it comes back as a follower. Then stop two brokers and see what changes for acks=all.
Why it exists
Why does the automatic handling need to be understood in this much detail? Because during an incident the most damaging actions are the well-meant ones. Restarting a second broker while the first is still catching up. Starting a large reassignment without a throttle on a cluster that is already short of capacity. Lowering min.insync.replicas to make producer errors stop. Enabling unclean leader election to get a partition back, and discovering later that acknowledged orders are gone. Each of these looks like a fix and each makes a recoverable situation worse.
A fixed order of work protects against this. It forces two questions before any change: what is the cluster already doing about this by itself, and does the action I am about to take reduce the number of in-sync replicas anywhere?
Availability first
- Get the partition serving again as fast as possible
- Accept that some acknowledged records may be lost
- Reasonable for metrics, clickstream, logs that are sampled anyway
- Mechanism: an unclean leader election
Consistency first
- Keep the partition offline until a replica with all committed data returns
- Accept an outage that lasts as long as the repair
- Required for orders, payments, anything reconciled against another system
- Mechanism: the default,
unclean.leader.election.enable=false
Neither side is correct in general. What matters is that the choice is made per topic, in advance where possible, and by someone who can say what the lost records were worth. The Kafka design documentation frames it the same way: a trade-off between availability and consistency that exists in every replicated system.
How it works inside
A broker crash, step by step
- Heartbeats stop. Each broker sends a heartbeat to the active controller every
broker.heartbeat.interval.ms(2 seconds). A crashed broker sends none. - The session expires. After
broker.session.timeout.ms(9 seconds) without a heartbeat, the controller fences the broker: it is still registered, but it may not lead partitions and is not offered to clients. - The controller rewrites partition state. For every partition with a replica on that broker it removes the broker from the ISR. For every partition the broker led, it elects a new leader, choosing a replica from the remaining ISR, and increments the leader epoch. These changes are records in the metadata log.
- Brokers apply the change. The new leaders start accepting produce and fetch requests. Followers begin fetching from them, truncating first if their log is ahead of what the new leader has.
- Clients find the new leader. A producer or consumer still talking to the dead broker gets a connection error or a request timeout; one talking to a broker that is no longer leader gets
NotLeaderOrFollowerException. Both are retriable and both make the client refresh its metadata and retry against the new leader.
From the client's side the gap is the detection time plus one metadata refresh, on the order of ten seconds with default settings. A producer with idempotence and the default delivery.timeout.ms of 2 minutes rides through it: records wait in its buffer and are sent, in order and without duplicates, to the new leader. A producer with a short delivery timeout or retries=0 reports failures instead. Consumers pause on the affected partitions and resume from their position, which is why a single broker failure should never appear as lost or duplicated data by itself.
A controlled shutdown (the broker process is stopped normally) skips the waiting. The broker tells the controller it is leaving, the controller moves its leaderships first, and only then does the process exit. controlled.shutdown.enable is true by default. The documentation notes two benefits: leadership moves within milliseconds per partition, and the logs are flushed so no recovery is needed at restart. This is why killing a broker with kill -9 is never an acceptable way to restart one.
When the broker returns
- Log recovery. After an unclean stop the broker validates the tail of each log before using it. On a broker with many partitions this takes time;
num.recovery.threads.per.data.dir(default 2) sets the parallelism and theremainingLogsToRecovermetric shows progress. After a controlled shutdown this step is skipped. - Registration and metadata catch-up. The broker registers with the controller and replays the metadata log to learn what changed while it was away. It stays fenced until it has caught up.
- Follower for everything. Once unfenced it is a follower for all its partitions, including those it used to lead. It truncates any records it holds beyond the new leader's history for that epoch, then fetches what it missed.
- Rejoining the ISR. When a replica has caught up to the leader's log end, the leader proposes adding it to the ISR. Under-replicated partitions fall back towards 0.
- Preferred leader election. The first broker in a partition's replica list is its preferred leader. With
auto.leader.rebalance.enable=true(the default) the controller checks everyleader.imbalance.check.interval.seconds(300) and hands leadership back to preferred replicas that are in the ISR. Until then the returned broker serves no client traffic as a leader.
When no in-sync replica is left
If every replica in a partition's ISR is on a stopped broker, the partition is offline. A replica that is running but was not in the ISR may be missing committed records, so by default it is not allowed to lead. From Kafka 4.0 the controller can also track eligible leader replicas (ELR, KIP-966): replicas that left the ISR while the partition was already below min.insync.replicas. Because the high watermark cannot advance in that state, such a replica is known to hold everything that was committed and is safe to elect. ELR is enabled by default on clusters created with 4.1 or later, and can be enabled on upgraded clusters. The election order is: a member of the ISR; otherwise an unfenced member of the ELR; otherwise wait.
An unclean leader election breaks the wait by electing a replica that is neither. Its log becomes the truth for the partition. Every record it lacks is gone, including records that producers were told were committed, and when the old leader returns it truncates its log to match. Consumers that had already read those records now hold offsets beyond the end of the new log. unclean.leader.election.enable is false by default and can be set per topic. Setting it dynamically does not act at once: the controller applies it on a periodic check, so for an immediate effect on chosen partitions the documentation points to kafka-leader-election.sh --election-type unclean.
A full disk
When a write to a log directory fails, the broker marks that directory offline (OfflineLogDirectoryCount), stops serving the replicas in it and reports this to the controller, which elects other leaders. A broker with several directories in log.dirs keeps running on the remaining ones. A broker whose only directory, or every directory, has failed shuts itself down. What makes a full disk harder than a crash is that restarting does not help: there is still no space, and recovery itself needs to write.
Space is released only by deleting whole log segments, which happens when retention says so. The safe ways out are to add capacity to the volume, or, while the brokers are still running, to shorten retention on the largest topics with kafka-configs.sh and let the broker delete old segments itself. Deleting files by hand from a partition directory is how a full disk turns into data loss. The Break card Too full to empty shows the same trap in a different queueing system: a store so full that the operation that would drain it cannot run.
A slow broker
Typical causes are a degrading disk, a noisy neighbour, a network interface dropping packets, or long garbage collection pauses. The signs are indirect. Partitions led by the slow broker show high produce latency, with the time in LocalTimeMs on that broker. Partitions it follows show high RemoteTimeMs on their leaders, and it keeps dropping out of ISRs and rejoining. Producers see timeouts for some partitions and not others. Consumer lag grows on the partitions it leads. If you can see that one broker is the common factor and it is not recovering, stop it with a controlled shutdown. The cluster then handles a problem it knows how to handle.
Losing the controller quorum
If fewer than a majority of controllers are running, there is no active controller. Brokers keep their last metadata, existing leaders keep accepting writes, and consumers keep reading. But nothing that needs a metadata change can happen: no leader election, no ISR shrink or expansion, no topic creation, no broker registration. A cluster in this state looks healthy to its clients and has no ability to respond to the next failure. Restoring the quorum comes before every other task. Do not restart brokers while it is down, because a restarted broker cannot register and will not come back.
Controller quorum
Simulation, not a real clusterFail a controller and watch a new one elected by majority, then see what happens when a majority is not available.
Step 1 · The pattern
One controller runs the cluster
Brokers store the data, and one active controller decides which broker leads each partition. Every broker follows the same controller, so every client gets the same map of the cluster.
- write
- confirmed
An incident-response sequence
- Stabilise. Stop changing things. Pause deployments, restarts and reassignments that are in flight. Confirm there is an active controller. If one broker is identified as slow or flapping, stop it cleanly. Do not start anything that removes another replica.
- Assess, worst first. List offline partitions, then under-min-ISR partitions, then under-replicated ones, and work out from the
ReplicasandIsrcolumns which brokers are missing. This tells you who is affected (which topics, so which services) and what would fix it (which broker has to return). - Decide about data deliberately. If a replica with the committed data can be brought back in acceptable time, wait for it. An unclean election or a lowered
min.insync.replicasis a decision for the owner of the data, taken per topic, and recorded with the time and the reason. - Communicate. Tell the teams that produce to and consume from the affected topics what is refused, what is delayed and what, if anything, may be lost. Producers need to know whether their buffered or failed sends will need replaying; consumers need to know whether to expect a backlog.
- Recover fully. Bring replicas back, replace what cannot be repaired with a throttled reassignment, wait for under-replicated partitions to reach 0, and rebalance leadership.
- Review. Write down the timeline, which signal should have fired earlier, and which manual step should become a check in tooling.
Partition reassignment
Kafka does not move replicas by itself. If a broker is gone for good, its partitions stay under-replicated until you change their replica lists. kafka-reassign-partitions.sh does this in three mutually exclusive modes: --generate proposes a plan for a list of topics and target brokers, --execute starts a plan given as JSON, and --verify reports whether each partition has finished. --list shows reassignments in progress and --cancel stops one.
Mechanically, the controller adds the new replica to the partition as a follower. It copies the whole partition from the leader. When it has caught up and joined the ISR, the replica being removed is dropped and its data deleted. The copy is the expensive part: it reads old segments from the leader's disk and uses network on both sides. --throttle (bytes per second) caps that traffic so it cannot starve producers and consumers. Two rules from the documentation matter. The throttle must be higher than the rate at which the partition is being written, or the new replica never catches up. And --verify must be run after completion, because it is what removes the throttle. A throttle left in place slows ordinary replication later.
Architecture
Drawing diagram…
A state diagram for one partition. Healthy becomes under-replicated when one replica is lost and returns when it rejoins the ISR. A second loss makes it under min ISR, where acks=all writes are refused but reads work. Losing the last in-sync replica makes it offline. It leaves offline either cleanly, when an in-sync or eligible replica returns, or through an unclean leader election, which leads to a data loss state before other replicas truncate and the partition is under-replicated again.
stateDiagram-v2
[*] --> Healthy
Healthy --> UnderReplicated: one replica lost or lagging
UnderReplicated --> Healthy: replica catches up and rejoins the ISR
UnderReplicated --> UnderMinIsr: second replica lost
UnderMinIsr --> UnderReplicated: one replica rejoins the ISR
UnderMinIsr --> Offline: last in-sync replica lost
Offline --> UnderMinIsr: an in-sync or eligible replica returns
Offline --> DataLoss: unclean leader election
DataLoss --> UnderReplicated: other replicas truncate to the new leader
note right of UnderReplicated
Reads and writes work
Margin is reduced
end note
note right of UnderMinIsr
acks=all writes refused
Reads still work
end note
note right of Offline
No leader
Nothing works on this partition
end note- Each step down the left side is one more replica lost. Kafka handles the first without help, refuses durable writes at the second, and stops at the third.
- There are two ways out of the offline state and they are very different. The clean one waits for a replica that has the data. The other accepts whatever the chosen replica has.
- The diagram is per partition. During one incident different partitions are in different states, which is why the first assessment step is to list them.
- Most actions taken in a hurry (restarting another broker, an unthrottled reassignment) push partitions down this diagram, not up.
Message flow
Drawing diagram…
A sequence diagram. A producer writes to the leader on broker 1 and is acknowledged. Broker 1 crashes and the next batch gets no response and stays buffered. After nine seconds without a heartbeat the controller fences broker 1, shrinks the ISR, makes broker 2 the leader and increments the epoch. The producer refreshes metadata, learns the new leader and retries the batch with the same sequence numbers, which is acknowledged. Broker 1 restarts, registers, fetches as a follower from broker 2, is added back to the ISR, and later a preferred election returns leadership to it.
sequenceDiagram participant P as Order Service producer participant B1 as Broker 1 leader participant C as Active controller participant B2 as Broker 2 follower P->>B1: Produce to orders.events-0 with acks=all B1-->>P: Acknowledged Note over B1: Broker 1 crashes P->>B1: Produce next batch Note over P: No response, batch stays buffered C->>C: No heartbeat from broker 1 for 9 seconds C->>C: Fence broker 1, ISR becomes 2 and 3, leader is 2, epoch plus one C-->>B2: Metadata record, you lead orders.events-0 P->>B2: Metadata request after timeout B2-->>P: Leader of orders.events-0 is broker 2 P->>B2: Retry the buffered batch with the same sequence numbers B2-->>P: Acknowledged Note over B1: Broker 1 restarts B1->>C: Register and catch up on metadata B1->>B2: Fetch as follower, truncate then copy B2->>C: Propose ISR 2 and 3 and 1 C->>C: Later, preferred election returns leadership to broker 1
- The batch sent to the dead broker is not lost and not duplicated. It waits in the producer and is retried with the same sequence numbers, so the idempotent producer and the new leader agree on what has been written.
- Between the crash and the fencing, about nine seconds with defaults, the partition has a leader on paper that is not answering. This window is the main part of what clients experience.
- The returning broker does not resume where it stopped. It learns from the metadata that it is now a follower, and brings its log in line with the new leader before copying.
- Leadership returns to broker 1 only after it is back in the ISR and the periodic preferred election runs. Until then broker 2 carries extra leader load.
In a real system
A typical incident at a payments company. The cluster has three brokers, one per availability zone, and three dedicated controllers. Topic payments.events has 12 partitions, replication factor 3 and min.insync.replicas=2, keyed by payment id. The Payment Service produces with acks=all; the Ledger Service and the Notification Service consume it in separate groups.
| Time | Event | State of payments.events | Decision |
|---|---|---|---|
| 02:10 | Broker 3's data volume starts returning I/O errors. Its log directory goes offline and the broker shuts down. | Under-replicated: ISR is 1,2 on all 12 partitions. Writes continue. | A ticket is opened for the morning. Reasonable: no client is affected. |
| 08:45 | A new volume is attached and broker 3 starts with an empty disk. It begins copying all 12 partitions from the leaders. | Still under-replicated. Leaders on 1 and 2 are now also serving a full copy. | Nobody limits or watches the catch-up traffic. |
| 09:20 | Morning traffic peaks. Broker 2, serving clients and the copy, has request handler idle below 10%. An engineer restarts it, reading the alert as a hung broker. | Under min ISR: ISR is 1 only. Every acks=all write is refused. | The wrong action: it removed an in-sync replica while another was missing. |
| 09:21 | Payment Service returns errors to checkout. Page fires on under-min-ISR partitions. | Under min ISR. | Incident declared. One person takes command, one handles communication. |
| 09:24 | Someone proposes setting min.insync.replicas=1 on the topic so that writes resume. | Under min ISR. | Declined. Broker 1 would be the only copy of every new payment event. Broker 2 is restarting and is minutes away. |
| 09:31 | Broker 2 finishes its controlled restart and catches up on nine minutes of nothing: no writes were accepted. | Under-replicated: ISR is 1,2. Writes resume. | Checkout recovers. Status update sent with the cause as currently understood. |
| 11:50 | Broker 3 completes its copy and rejoins every ISR. | Healthy. | Incident closed only now. |
The review found three things worth changing, none of which was the disk. A broker that returns empty should have its catch-up limited so that it cannot overload the leaders it copies from. An overloaded broker and a hung broker look alike on a single graph, so the runbook now says to check under-replicated partitions before restarting anything, and to never restart a broker while that number is not 0 unless the broker is the cause. And the question about lowering min.insync.replicas now has a written answer per topic, agreed with the business owners in advance, so it is not argued at 09:24.
Java and Spring Boot
The commands for assessing an incident, the reassignment that replaces a lost broker, and the producer settings that decide whether an application rides through a leader change or reports errors. Sample output is illustrative, not captured from a real cluster.
BS=broker1.example.com:9092# 1. Is there an active controller? Without one nothing below can change.bin/kafka-metadata-quorum.sh --bootstrap-server $BS describe --status# Illustrative output:# LeaderId: 3001# LeaderEpoch: 14# HighWatermark: 982113# MaxFollowerLag: 0# CurrentVoters: [{"id": 3000, ...}, {"id": 3001, ...}, {"id": 3002, ...}]# 2. Worst first: partitions with no leaderbin/kafka-topics.sh --bootstrap-server $BS --describe --unavailable-partitions# 3. Partitions refusing acks=all writesbin/kafka-topics.sh --bootstrap-server $BS --describe --under-min-isr-partitions# Illustrative output:# Topic: payments.events Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1# Topic: payments.events Partition: 1 Leader: 1 Replicas: 2,3,1 Isr: 1# 4. Everything with reduced marginbin/kafka-topics.sh --bootstrap-server $BS --describe --under-replicated-partitions# 5. Is a reassignment already running? (It would explain extra load.)bin/kafka-reassign-partitions.sh --bootstrap-server $BS --listWhy it is written this way
- The order is by severity. An offline partition and a missing controller change what you do next; under-replication alone usually means wait and watch.
- In each output line, the broker ids present in
Replicasand absent fromIsrare the ones to investigate. Here brokers 2 and 3 are missing everywhere. - If brokers are unreachable, the quorum tool also accepts
--bootstrap-controllerwith a controller address. - On 4.x the describe output may also print
ElrandLastKnownElrfields. A replica listed underElrcan be elected without data loss.
BS=broker1.example.com:9092# Broker 3 is not coming back. Broker 4 is new and empty.# Move every replica that was on 3 to 4, keeping the other replicas where they are.cat > replace-broker-3.json <<'JSON'{"version":1,"partitions":[ {"topic":"payments.events","partition":0,"replicas":[1,2,4]}, {"topic":"payments.events","partition":1,"replicas":[2,4,1]}, {"topic":"payments.events","partition":2,"replicas":[4,1,2]}]}JSON# Start it with a throttle of 30 MB/s. The command prints the current assignment:# save that output, it is the rollback plan.bin/kafka-reassign-partitions.sh --bootstrap-server $BS \ --reassignment-json-file replace-broker-3.json --execute --throttle 30000000# Progress. Run it again until every partition is completed.# When all are complete, this same command removes the throttle.bin/kafka-reassign-partitions.sh --bootstrap-server $BS \ --reassignment-json-file replace-broker-3.json --verify# Illustrative output:# Status of partition reassignment:# Reassignment of partition [payments.events,0] is completed# Reassignment of partition [payments.events,1] is still in progress# Reassignment of partition [payments.events,2] is still in progress# Too slow and the cluster has headroom? Raise the throttle on the running reassignment.bin/kafka-reassign-partitions.sh --bootstrap-server $BS \ --reassignment-json-file replace-broker-3.json --execute --additional --throttle 60000000# Hurting live traffic? Stop it.bin/kafka-reassign-partitions.sh --bootstrap-server $BS \ --reassignment-json-file replace-broker-3.json --cancelWhy it is written this way
- Only the position that held broker 3 changes in each replica list. Keeping the order of the others keeps the preferred leaders where they were.
- The tool cannot work out a replacement plan by itself:
--generatespreads whole topics over a broker list, which is a different operation. For a one-for-one replacement the JSON is written by hand or by a script. - A throttle below the partition's incoming write rate means the new replica never catches up. Compare it with
BytesInPerSecfor the topic before choosing a number. --verifyis not optional housekeeping. It is the step that clears the throttle configuration from brokers and topics.- During an incident, reassignment is a recovery step for after clients are served again. It adds load, so do not start it while partitions are under min ISR unless it is the only way to regain a replica.
BS=broker1.example.com:9092# LAST RESORT, per partition, after a recorded decision.# Elects a replica that may be missing committed records. Those records are lost# for good and consumers may hold offsets past the new end of the log.# First check what the candidates are: prefer waiting for a replica in Isr or Elr.bin/kafka-topics.sh --bootstrap-server $BS --describe --topic clickstream.raw# One named partition onlybin/kafka-leader-election.sh --bootstrap-server $BS \ --election-type unclean --topic clickstream.raw --partition 4# Afterwards, when brokers are back: return leadership to preferred replicasbin/kafka-leader-election.sh --bootstrap-server $BS \ --election-type preferred --all-topic-partitionsWhy it is written this way
- The tool performs an unclean election only for partitions that currently have no leader, so it cannot displace a healthy one.
- Naming the topic and partition keeps the blast radius to what was decided.
--all-topic-partitionswith the unclean type applies the trade to every leaderless partition in the cluster. - The example uses a clickstream topic on purpose. For orders or payments the usual answer is to keep waiting and repair the broker that has the data.
- After an unclean election, tell the consuming teams which partition and roughly which time window is affected. They may need to reset offsets or reconcile against another source.
spring: kafka: bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS} producer: acks: all properties: enable.idempotence: true # Total time a record may spend buffered and retrying before send() fails. # Must cover failure detection plus a metadata refresh, with room to spare. delivery.timeout.ms: 120000 request.timeout.ms: 30000 # How long send() may block when the buffer is full or metadata is missing. max.block.ms: 60000Why it is written this way
- These are the kafka-clients defaults written out, so the incident behaviour is a visible decision and not an accident of the library version.
- With a 2 minute delivery timeout, a single broker failure of about 10 seconds is absorbed by retries. A partition that is under min ISR for 10 minutes is not: sends fail after 2 minutes, and the application must have a plan for that, for example the outbox pattern.
max.block.msis the setting that decides how long request threads hang when the cluster is unreachable. If producing happens on a web request thread, a full minute of blocking can exhaust the thread pool, so some services lower it and fail fast.- Idempotence is what makes the retry after a leader change safe: the new leader discards a batch it already has. It covers the producer's own retries only, not an application that sends the same event twice.
Configuration
The settings that decide how quickly a failure is detected, what the cluster does about it, and what a returning broker does. Defaults are from the Apache Kafka 4.4 configuration reference.
broker.session.timeout.msDefault 9000 (9 seconds)
- What it does
- How long the controller waits without a heartbeat before fencing a broker and moving its leaderships.
- When to change it
- The main part of client-visible failover time after a crash. Lowering it gives faster failover and more false fencing during GC pauses or network blips. Change it only with evidence.
broker.heartbeat.interval.msDefault 2000 (2 seconds)
- What it does
- How often a broker sends a heartbeat to the active controller.
- When to change it
- Keep it well below the session timeout so that a few lost heartbeats do not fence a healthy broker.
replica.lag.time.max.msDefault 30000 (30 seconds)
- What it does
- A follower that has not caught up to the leader's log end for this long is removed from the ISR.
- When to change it
- This is how a slow follower is handled, as opposed to a dead broker. The producer's
request.timeout.msis documented as needing to be larger than this value.
unclean.leader.election.enableDefault false
- What it does
- Whether a replica outside the ISR may be elected leader as a last resort, at the risk of losing committed records. Broker default and topic-level setting.
- When to change it
- Leave false for data that must not be lost. Consider true per topic for data where availability matters more. A dynamic change takes effect on a periodic controller check, not at once.
min.insync.replicasDefault 1
- What it does
- The smallest ISR size at which
acks=allwrites are accepted. - When to change it
- 2 with replication factor 3 tolerates one failure with no loss of durability. Lowering it during an incident restores writes by allowing a single copy: a decision for the data owner, not a reflex.
controlled.shutdown.enableDefault true
- What it does
- On a normal stop, the broker asks the controller to move its leaderships away before the process exits.
- When to change it
- Leave on. Make sure your process manager gives the broker enough time to finish before it sends a hard kill.
auto.leader.rebalance.enableDefault true
- What it does
- The controller periodically moves leadership back to each partition's preferred replica when that replica is in the ISR.
- When to change it
- Leave on unless you run preferred elections yourself with
kafka-leader-election.sh --election-type preferred. Without either, leaders pile up on the brokers that did not restart.
leader.imbalance.check.interval.secondsDefault 300
- What it does
- How often the controller runs the check behind
auto.leader.rebalance.enable. - When to change it
- Explains why a returned broker leads nothing for up to five minutes after it is back in sync. Rarely worth changing.
num.recovery.threads.per.data.dirDefault 2
- What it does
- Threads per log directory used for log recovery at startup and for flushing at shutdown. The default was 1 before Kafka 4.0.
- When to change it
- Raise it on brokers with many partitions and fast disks to shorten restart after a crash.
delivery.timeout.ms (producer)Default 120000 (2 minutes)
- What it does
- Upper bound on the time from
send()returning to success or failure being reported, including retries. - When to change it
- Must comfortably exceed failover time. If your service cannot wait that long, lower it and handle the failure explicitly; do not disable retries.
Hands-on exercise
Troubleshoot: three brokers are up, and the topic is still offline
Work from an incident channel's evidence to the cause of an outage that survives restarting brokers, decide what to do about it, then run the same failure in the broker-failure lab.
Steps · 0 of 7 done
Reproduce the outage
Simulation · not a real clusterThree brokers, orders.events with 3 partitions, replication factor 3, min.insync.replicas 2. Follow the incident timeline from the exercise steps.
What you should see
The last in-sync replica of every partition is broker 1: the Isr column says so, and it is the one broker still down. Broker 3 left the ISR at 14:02 and missed the records produced until 14:06 (5 per partition in the widget, visible as a lower log end offset). In the widget, restarting brokers 3 and 2 leaves all three partitions offline, and the event log explains that making either leader would be an unclean election. Electing broker 3 would discard those acknowledged records, so for orders the answer is to keep the topic offline and bring broker 1 back; when it restarts, every partition gets a leader, the others catch up, and the preferred election spreads leadership again. One difference from a real 4.x cluster: broker 2 left the ISR when the partition was already at its minimum, and nothing could be committed after that (the second Produce was rejected), so with eligible leader replicas enabled the controller may list broker 2 under Elr and elect it cleanly when it returns. The widget does not model ELR. On a real cluster, read the Elr field before deciding anything.
What can go wrong
Restarting brokers to see if it helps
- Symptom
- An incident with producer errors on some partitions becomes one with errors on all of them shortly after an engineer restarts a broker that looked unhealthy.
- Cause
- The cluster was already under-replicated. The restarted broker held the second in-sync replica of many partitions, so stopping it took them below
min.insync.replicas. - Fix
- Check under-replicated and under-min-ISR partitions before stopping any broker. Stop a broker during an incident only when it is identified as the cause.
Unclean election as a reflex
- Symptom
- After an outage, downstream services report missing events, or consumers fail with offset-out-of-range errors and reset to the earliest or latest offset.
- Cause
- A replica outside the ISR was made leader to end the outage. Records it had not replicated were discarded, including acknowledged ones.
- Fix
- Keep
unclean.leader.election.enable=falsefor durable topics. Decide per topic in advance who may authorise an unclean election, and checkIsrandElrfor a clean candidate first.
Lowering min.insync.replicas to stop the errors
- Symptom
- Producer errors stop during the incident. Later, a second broker failure loses the events written in between.
- Cause
- With
min.insync.replicas=1,acks=allis satisfied by the leader alone. Every write during that period existed on one disk. - Fix
- Treat it as a deliberate, time-boxed choice made by the data owner, recorded in the incident log, and reverted as soon as a second replica is in sync.
Unthrottled or forgotten reassignment
- Symptom
- Produce latency and consumer lag rise across the cluster during a reassignment; or, weeks later, followers lag for no obvious reason.
- Cause
- Copying whole partitions competes with client traffic for disk and network. In the second case a throttle was set and
--verifywas never run, so ordinary replication is still rate-limited. - Fix
- Always pass
--throttle, sized above the topic's write rate and below the spare capacity. Run--verifyuntil complete, and checkkafka-configs.sh --describe --entity-type brokersfor leftover throttle settings.
Declaring recovery when the process is up
- Symptom
- The incident is closed. An hour later a routine deployment restarts another broker and writes are refused again.
- Cause
- The returned broker was still catching up and was not in any ISR. The cluster had no more margin than during the incident.
- Fix
- Close on under-replicated partitions at 0 and leadership rebalanced, not on process status. Block maintenance automatically while under-replicated partitions exist.
Freeing disk space by deleting files
- Symptom
- After a full-disk incident a broker fails to start, or partitions report missing or corrupt segments.
- Cause
- Segment or index files were removed by hand from a partition directory while the broker's own view of the log still included them.
- Fix
- Add capacity, or reduce
retention.ms/retention.byteson the largest topics withkafka-configs.shand let the broker delete segments. Restore the original retention afterwards.
How to troubleshoot
Producers get NotLeaderOrFollowerException or timeouts for a few seconds, then recover
Check
`IsrShrinksPerSec` on the surviving brokers and `FencedBrokerCount` on the controller at that time; the controller log for the broker being fenced; `kafka-topics.sh --bootstrap-server … --describe --topic orders.events`
- How to read it
- A normal leader change after a broker stop. If it recovers within the producer's delivery timeout and no send failed, nothing is wrong with the clients. Find out why the broker stopped.
Producers fail with NotEnoughReplicasException
Check
`kafka-topics.sh --bootstrap-server … --describe --under-min-isr-partitions`
- How to read it
- Two replicas of those partitions are missing or lagging. Compare
ReplicaswithIsrto find which brokers. Getting one of them back in sync restores writes; consumers were never blocked.
A partition has no leader although some of its replicas are running
Check
`kafka-topics.sh --bootstrap-server … --describe --unavailable-partitions`, reading the `Isr` and `Elr` columns
- How to read it
- The running replicas are not in the ISR (or ELR), so electing them is not safe. The brokers listed in
Israre the ones that hold the committed data: bring one of those back.
A restarted broker has been up for a long time and partitions are still under-replicated
Check
`remainingLogsToRecover` on that broker; `kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica`; `ReplicationBytesInPerSec`; leftover throttles with `kafka-configs.sh --bootstrap-server … --describe --entity-type brokers`
- How to read it
- Non-zero logs to recover: it is still validating its logs after an unclean stop. Lag falling steadily: it is copying and needs time. Lag flat or rising: it cannot fetch faster than the leaders are written to, often because of a forgotten replication throttle or a slow disk.
Latency and timeouts on some partitions only, all brokers running
Check
`kafka.network:type=RequestMetrics,name=LocalTimeMs,request=Produce` per broker; which broker leads the affected partitions; `IsrShrinksPerSec`; host disk latency and network errors
- How to read it
- If one broker leads all the affected partitions and its local time stands out, it is a slow broker. It will not be fenced while it heartbeats. A controlled shutdown moves its leaderships to healthy replicas.
A broker stopped and will not stay up after restart
Check
Its server log for a log directory failure or `Shutdown broker because all log dirs in … have failed`; free space on the volumes in `log.dirs`; `OfflineLogDirectoryCount` before it stopped
- How to read it
- A failed or full log directory. Restarting cannot fix it. Repair or extend the volume first; if the disk is lost, start the broker empty and let it copy, with catch-up traffic limited.
Topic creation hangs, a stopped broker's partitions never get new leaders, a restarted broker never becomes active
Check
`kafka-metadata-quorum.sh --bootstrap-controller controller1.example.com:9093 describe --status`; `ActiveControllerCount` summed over the controllers
- How to read it
- No active controller: the quorum lacks a majority. Existing leaders still serve, which hides the problem. Restore the controllers before touching any broker.
Key takeaways
- One broker failure with replication factor 3 and min ISR 2 is handled by the cluster: about ten seconds of retries with default timeouts, no loss, no duplicates from an idempotent producer.
- A returned broker is a follower that must catch up. Recovery is complete when under-replicated partitions are 0 and leadership is rebalanced.
- A slow broker is not fenced and hurts clients for as long as it runs. Stopping it deliberately is a valid fix.
- Without a controller majority the cluster serves but cannot react. Restore the quorum first.
- An unclean leader election and a lowered
min.insync.replicasboth trade acknowledged data for availability. Make that choice per topic, on purpose, and write it down. - Kafka does not move replicas by itself. Use
kafka-reassign-partitions.shwith a throttle, and run--verifyto finish and remove it. - Incident order: stabilise, list offline and under-min-ISR partitions, decide about data deliberately, communicate, review.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
Topic orders.events: replication factor 3, min.insync.replicas=2. Broker 5 has been down for an hour for a hardware repair, so 40 partitions show ISR size 2. Broker 2's request handler idle ratio drops to 0.05 at peak and produce latency on its partitions is 800 ms. A colleague wants to restart broker 2 to clear it.
What happens to the partitions that have replicas on both broker 2 and broker 5 if broker 2 is restarted now?
Scenario 2 of 3
Partition clicks.raw-7 and partition payments.events-3 are both offline after a double failure. For each, describe shows Replicas 4,5,6 and Isr 6, with broker 6 needing a disk replacement that takes about three hours. Brokers 4 and 5 are running. clicks.raw feeds a dashboard with sampled analytics; payments.events feeds the ledger.
What is the sound decision?
Scenario 3 of 3
Broker 3 died permanently. You start a reassignment moving its 200 replicas to a new broker 7 with --throttle 10000000 (10 MB/s). After two hours --verify still reports most partitions in progress, and the replica lag for the busiest topic on broker 7 is growing. That topic receives about 25 MB/s from producers across its partitions.
What explains this, and what do you do?
References
References: Apache Kafka documentation: Basic Kafka operations (graceful shutdown, balancing leadership, reassignment, throttling) ↗ · Apache Kafka documentation: Design (replication, unclean leader election, availability and durability) ↗ · Apache Kafka documentation: Eligible Leader Replicas ↗ · Apache Kafka documentation: Broker configuration reference ↗ · Apache Kafka documentation: Producer configuration reference ↗ · Apache Kafka documentation: Operations, KRaft (quorum tool) ↗ · Apache Kafka documentation: Monitoring ↗ · KIP-966: Eligible Leader Replicas ↗ · KIP-455: Create an Administrative API for Replica Reassignment ↗ · Report a problem