Path 10 · lesson 3 of 6
Rolling upgrades and scaling a cluster
How to restart, upgrade, grow and shrink a Kafka cluster one broker at a time without refusing a write, and which steps cannot be undone.
What you will learn
- Restart every broker in a cluster in turn while keeping every partition writable with
acks=all - Name the check that must pass between two brokers and explain why a port or process check is not it
- Separate the two phases of an upgrade, new software and feature finalisation, and say which one can be rolled back
- Plan an upgrade path from a 3.x cluster to 4.x, including clusters still on ZooKeeper
- Add a broker and move partitions onto it under a throttle, and remove a broker cleanly
- Judge when increasing a topic's partition count is safe and what it breaks
Before you start
The concept
Almost every planned change to a Kafka cluster is the same operation repeated: take one broker out, change it, put it back, wait until it is fully in sync, move to the next. A configuration change, an operating system patch, a new Kafka version and a hardware replacement all use it. It works because of replication: with replication factor 3 and min.insync.replicas=2, one broker may be absent at any moment and every partition still has two in-sync replicas.
That sentence also states the limit. The procedure is safe only while at most one replica of any partition is out of sync. A rolling restart is a sequence of deliberate single-broker failures, and the previous lesson showed what the second concurrent failure does. So the whole skill is in the waiting: knowing what to wait for and refusing to continue until it is true.
| Operation | How it is done | What makes it safe | Reversible? |
|---|---|---|---|
| Restart or reconfigure brokers | Rolling restart, one broker at a time | No under-replicated partitions before each stop; controlled shutdown | Yes |
| Upgrade the Kafka software | Rolling restart with the new version | Same, and the cluster still speaks the old metadata version | Yes, by rolling back to the old software |
| Finalise the upgrade | kafka-features.sh upgrade --release-version … | Done only after the new software has run long enough to trust | Often not: depends on the metadata versions involved |
| Add a broker | Start it, then reassign partitions to it | Throttle on the data movement | Yes, by reassigning back |
| Remove a broker | Reassign everything away, stop it, unregister it | It holds no replicas when it stops | Yes, until it is unregistered and wiped |
| Increase a topic's partitions | kafka-topics.sh --alter --partitions | Nothing makes it safe for keyed data in flight | No: partitions cannot be reduced |
One at a time
Simulation · not a real clusterStop one broker and watch the Under-replicated counter. Restart it and wait for the counter to return to 0 before stopping the next. Then try stopping two at once and press Produce.
Why it exists
A cluster that could only be changed by stopping it would force a choice between running old software and scheduling outages. Kafka is the shared dependency of many services, so an outage window for Kafka is an outage window for all of them. Rolling operations exist so that the cluster's own maintenance is invisible to its clients: a producer sees a leader move and retries, a consumer sees the same and continues.
Upgrades add a second problem. During a rolling upgrade, brokers of two versions run side by side and must understand each other, and what they write to disk and to the metadata log must be readable by both. Kafka solves this by separating the software version from the feature level the cluster uses. New software starts out behaving like the old version. Only when every node runs the new software do you switch the cluster to the new feature level, as an explicit, separate step.
Phase 1: roll out the software
- Each broker is restarted with the new version
- The cluster keeps using the old
metadata.version - Mixed versions run together safely
- Rollback is another rolling restart with the old version
Phase 2: finalise
- One command raises
metadata.versionand other features for the whole cluster - New record formats and protocols come into use
- Old software may no longer be able to read the metadata
- Rollback is possible only if no metadata changes lie between the versions
Scaling has its own reason for care. Kafka never moves data by itself. A new broker receives nothing until you assign partitions to it, and moving a partition means copying all of it. On a busy cluster that copy competes with the traffic you are scaling for, which is why capacity should be added before it is urgent. The Break card One server too many is a sourced example, from a different streaming system, of a routine capacity addition that pushed a whole fleet over a limit.
How it works inside
The rolling restart procedure
- Pre-flight. The cluster must be healthy before you start: no under-replicated partitions, no offline partitions, an active controller, no reassignment running. If any of these fail, the maintenance does not begin.
- Stop one broker with a controlled shutdown. A normal stop signal, never a hard kill. The broker asks the controller to move its leaderships to other in-sync replicas first, then flushes its logs and exits. Clients see leader changes of a few milliseconds per partition and retry.
- Apply the change and start the broker. It registers, catches up on metadata, becomes a follower for all its partitions and fetches what it missed.
- Wait for the gate. Do not continue until under-replicated partitions are back to 0. That is the proof that the broker is in every ISR again.
- Repeat for the next broker. Many teams restart the broker hosting the fewest leaders first and take a short pause after each to watch client error rates.
- Finish. Leadership ends up concentrated on the brokers restarted earliest. The controller's automatic preferred leader election corrects this within its check interval, or you can run
kafka-leader-election.sh --election-type preferred --all-topic-partitions.
Controlled shutdown has a precondition stated in the documentation: it succeeds only if every partition on the broker has another live replica to take over. A topic with replication factor 1 has none, so its partitions are simply unavailable while its broker is down. Rolling operations assume replication factor 3 and min.insync.replicas=2 for everything that matters; with replication factor 2 and min.insync.replicas=2, every restart refuses writes.
Controllers are rolled the same way, one at a time, keeping a majority of the quorum running throughout. With three controllers that means never having two down together. A common practice is to restart the standby controllers first and the active one last, so that there is one controller election and not several. If a server runs in combined mode (process.roles=broker,controller) each restart takes out a broker and a voter at once, one reason the documentation advises against combined mode for critical deployments.
Software version, metadata.version and finalisation
In KRaft the cluster's feature level is itself a piece of replicated metadata. metadata.version says which record formats and inter-broker behaviour the cluster uses. It is finalised: stored in the metadata log, the same for every node, and changed only by an explicit command. It replaces the inter.broker.protocol.version setting that ZooKeeper-based clusters edited in each broker's properties file. Other features are versioned the same way, for example kraft.version and eligible.leader.replicas.version. kafka-features.sh describe shows, for each feature, the range the software supports and the level currently finalised.
The documented upgrade procedure is short. Upgrade the brokers one at a time: shut down, update the code, restart. Once all are on the new version, verify that the cluster's behaviour and performance meet expectations. Then finalise with bin/kafka-features.sh --bootstrap-server … upgrade --release-version <version>.
Finalising can also switch on behaviour, which is another reason to give it its own change window. The 4.0 notes give two examples: the new consumer group protocol (KIP-848) becomes available on the server when the upgrade to 4.0 is finalised, and so does the strengthened transaction protocol.
From 3.x to 4.x
Kafka 4.0 removed ZooKeeper mode. A 4.x broker cannot start against ZooKeeper, so there is no rolling upgrade from a ZooKeeper-based cluster straight to 4.x. The upgrade notes give the supported paths:
| Your cluster today | Path to 4.x |
|---|---|
KRaft mode, software and metadata.version at 3.3 or later | Rolling upgrade directly to 4.x, then finalise. |
| KRaft mode, older than 3.3 | Upgrade to 3.9 first, then to 4.x. |
| ZooKeeper mode | Upgrade to a bridge release (3.9 is the last one), migrate the cluster from ZooKeeper to KRaft there following the 3.9 documentation, and only then upgrade to 4.x. |
Two more things changed at 4.0 that an upgrade plan must check. The broker needs Java 17 or later (clients still run on Java 11). And settings, tools and metrics that existed only for ZooKeeper are gone: zookeeper.connect, the --zookeeper flag, inter.broker.protocol.version, log.message.format.version and several controller metrics. Configuration files and dashboards carried over from 3.x need a review before the first 4.x broker starts.
Client and broker compatibility
Kafka clients and brokers negotiate protocol versions per request, so a client does not have to match the broker version. Clients and brokers are upgraded independently and in either order, within limits. Kafka 4.0 removed the oldest protocol versions (KIP-896): brokers on 4.x require clients at version 2.1 or later, and 4.x Java clients require brokers at 2.1 or later. The compatibility page lists 3.x clients as fully compatible with 4.0 brokers.
In practice: a Spring Boot 3.5 service, which uses kafka-clients 3.9, works against a 4.x cluster without changes. What it cannot do is use features that need a newer client. Before upgrading brokers to 4.x, find every client older than 2.1, and check clients that are not the Apache Java client (librdkafka-based and others) against their own compatibility notes, since KIP-896 affects them through the protocol versions they send.
Adding brokers
A new broker gets a unique node.id, has its storage formatted with the cluster's id, and is started. It joins, and then does nothing, because no partition is assigned to it. Only topics created afterwards may place replicas there. To use it, you reassign existing partitions with kafka-reassign-partitions.sh: --generate proposes an even spread of the topics you name over the broker list you give, --execute starts it, --verify reports completion. The tool does not look at partition sizes or traffic, so which topics to move is your decision.
For each moved partition the new broker is added as a follower, copies the whole partition, joins the ISR, and then one old replica is deleted. Always pass --throttle. The limit must be above the write rate of the partitions being moved, or the new replica never catches up, and --verify must be run at the end because it removes the throttle. Move in batches (a few topics at a time): a smaller plan finishes sooner, is easier to cancel, and keeps the number of partitions with an extra replica small.
Removing a broker
Decommissioning is the reverse, with one extra step at each end. The 4.4 documentation describes it as: cordon the broker so that no new replicas are placed on it (cordoned.log.dirs set to * for that broker; a recent setting, so check that your version has it), reassign every replica it holds to other brokers, shut it down once it holds none, and unregister it with kafka-cluster.sh unregister --id … so that the controller forgets it. The tool cannot generate a decommissioning plan for you. Before starting, check that the remaining brokers have the disk and network capacity for what they are about to receive, and that enough brokers remain for the replication factor.
Increasing partitions
kafka-topics.sh --alter --topic … --partitions N adds partitions at once. It is the cheapest scaling command to type and the one with the most lasting effects, which the documentation lists:
- Keys move. The default partitioner chooses a partition from the hash of the key and the partition count. With a new count, the same key can map to a different partition. Existing records stay where they are, so records for one key now sit in two partitions and their relative order is no longer guaranteed across that boundary.
- Kafka does not redistribute existing data. Old partitions keep their backlog; new ones start empty.
- It cannot be undone. Kafka does not support reducing the number of partitions.
- Consumers can miss the first records. A group with no committed offset for the new partition starts where
auto.offset.resetsays. Withlatest, records produced before the consumer discovers the partition are skipped. Recent client versions acceptby_duration:<duration>for this case; with older clients,earliestavoids the gap at the cost of nothing, since the partition is new. - Clients notice late. Producers and classic-protocol consumers learn of new partitions on a metadata refresh (
metadata.max.age.ms, 5 minutes by default). - Never on internal topics. Changing the partition count of
__consumer_offsets,__transaction_stateor other internal topics breaks the coordinator mapping.
So it is safe for topics without keys, and for keyed topics only when nobody depends on per-key order across the change or the topic can be drained first. For a keyed topic that needs more partitions and strict ordering, the usual route is a new topic with the target count and a controlled migration. Choosing a partition count with room to grow is covered in Partition sizing and parallelism.
Rolling out configuration changes
Broker settings differ in how they change. The configuration reference gives each an update mode: read-only settings need a broker restart, so a rolling restart; per-broker and cluster-wide settings can be changed on a running cluster with kafka-configs.sh. Topic settings such as retention are always dynamic. Prefer the dynamic route where it exists, and change one broker or one topic first.
Client settings change by redeploying the application, and for consumers every restarted instance leaves and rejoins its group. Roll out one instance first and compare its metrics with the others. Restart consumers a few at a time, and use static membership or the cooperative assignor so that a rolling deployment does not become a series of full rebalances; see Consumer rebalancing. Changes that alter what is written (compression type, a serializer, a partitioner or the key) deserve the most care, because every consumer of the topic will read the result.
Architecture
Drawing diagram…
A flowchart. A pre-flight check leads to a decision: if the cluster is not healthy, stop. Otherwise one broker is shut down in a controlled way, upgraded and started. A loop waits until under-replicated partitions are back to zero, then repeats for the next broker. When all nodes run the new software with the old metadata version, the cluster is observed for days. If a problem is found, roll back with the old version. If not, finalise with the features tool, after which the new feature level is in use and the step is treated as one-way.
flowchart TD
A["Pre-flight: under-replicated is 0, controller active, no reassignment"] --> B{"Healthy?"}
B -->|"no"| X["Stop. Fix the cluster first"]
B -->|"yes"| C["Controlled shutdown of one broker"]
C --> D["Install new version, start broker"]
D --> E{"Under-replicated back to 0?"}
E -->|"not yet"| W["Wait and watch catch-up"]
W --> E
E -->|"yes"| F{"More brokers?"}
F -->|"yes"| C
F -->|"no"| G["All nodes on new software, old metadata.version"]
G --> H["Observe for days: errors, latency, lag"]
H --> I{"Problem found?"}
I -->|"yes"| R["Roll back: rolling restart with old version"]
I -->|"no"| J["Finalise: kafka-features upgrade"]
J --> K["New feature level in use. Treat as one-way"]- The loop in the middle is the rolling restart. Its only exit is the under-replicated gate, which is why a slow catch-up makes the upgrade longer and not less safe.
- Everything above the finalise box can be undone by repeating the loop with the old binaries, because the cluster has not yet written anything the old version cannot read.
- The observation period is drawn as its own step on purpose. Problems that matter (a latency regression at peak, a client that misbehaves against the new brokers) often need a full traffic cycle to show.
- Finalising is a single command against the whole cluster, with no rolling and no gate. All the care goes into deciding when to run it.
Message flow
Drawing diagram…
A sequence diagram. The operator script confirms there are no under-replicated partitions, then stops broker 1. Broker 1 requests a controlled shutdown, the controller moves its leaderships to in-sync replicas such as broker 2, and tells broker 1 it may exit. The producer refreshes metadata and writes to the new leader with two in-sync replicas. The script starts broker 1 with the new version, it registers, fetches the missed records as a follower and is added back to the ISR. The script checks again, finds no under-replicated partitions, and continues with broker 2.
sequenceDiagram participant O as Operator script participant B1 as Broker 1 participant C as Active controller participant B2 as Broker 2 participant P as Producer O->>B2: Describe under-replicated partitions B2-->>O: None O->>B1: Stop with a normal signal B1->>C: Request controlled shutdown C->>C: Move leaderships of broker 1 to in-sync replicas C-->>B2: You lead orders.events-0 C-->>B1: Safe to shut down B1->>B1: Flush logs and exit P->>B2: Metadata refresh then produce to the new leader B2-->>P: Acknowledged, ISR is 2 and 3 O->>B1: Start with the new version B1->>C: Register and catch up on metadata B1->>B2: Fetch missed records as follower B2->>C: Propose ISR 2 and 3 and 1 O->>B2: Describe under-replicated partitions B2-->>O: None Note over O: Gate passed, continue with broker 2
- The same check opens and closes the step. It is asked of the cluster, not of the broker being restarted, because a broker cannot judge whether it is in sync.
- Leadership moves before the process exits, so producers are redirected by a metadata refresh and do not wait for a session timeout as they would after a crash.
- While broker 1 is down, writes with
acks=allsucceed with an ISR of two. That is the margin being spent, and why nothing else may be down at the same time. - The broker's return to the ISR is proposed by the partition leaders once it has caught up. How long that takes depends on how much was written while it was away.
In a real system
A typical incident during an upgrade from Kafka 3.9 to 4.x at a logistics company. Six brokers, three dedicated controllers, already on KRaft. Topic shipments.events has 48 partitions, replication factor 3 and min.insync.replicas=2, keyed by shipment id. The upgrade is automated with a script that had been used for configuration restarts for a year.
| Time | Event | What the tools showed (illustrative) | The flaw |
|---|---|---|---|
| 21:00 | Script starts. For each broker: stop, install, start, wait until port 9092 accepts connections, sleep 60 seconds, next. | Under-replicated partitions 0 at the start. | The gate is a port check and a timer. |
| 21:04 | Broker 1 back. It had been down 3 minutes and catches up in 20 seconds. | Under-replicated back to 0 well inside the 60 seconds. | The timer happened to be long enough, as it had been for a year. |
| 21:09 | Broker 2 is stopped by a hard kill: the service unit's stop timeout was shorter than the controlled shutdown needed. | Server log on restart shows log recovery running for every partition. | No controlled shutdown, so leaders moved only after the session timeout and recovery was needed. |
| 21:11 | Broker 2's port opens. It is still recovering logs and has not begun to fetch. The 60 seconds pass. | Under-replicated 410 and not falling. | Running is not in sync. |
| 21:12 | Script stops broker 3. | Under-min-ISR 137. Producers of shipments.events log NotEnoughReplicasException. | Two replicas of 137 partitions are out at once. |
| 21:26 | Broker 3 is back and in sync; broker 2 finishes recovery a few minutes later. | Under-min-ISR 0, then under-replicated 0. | Fourteen minutes of refused writes, caused entirely by the procedure. |
| 21:40 | The script completes and, as its last line, runs kafka-features.sh upgrade. | Finalised metadata.version at the new level. | No observation period. |
| Next day | A consumer latency regression appears at peak. The team wants to roll back to 3.9 while investigating. | The upgrade notes for 4.0 state that metadata downgrade is not supported. | Rollback was given away twelve hours earlier for no benefit. |
Three changes came out of the review. The gate between brokers became the cluster's own answer to whether any partition is under-replicated, with a timeout that stops the script and pages a person. The service unit's stop timeout was raised so that controlled shutdown can finish, and the script checks the broker's log for a clean shutdown. And finalisation was removed from the script: it is now a separate change, made by a person, no sooner than a week after the last broker is upgraded.
Java and Spring Boot
A restart script with the right gate, the two-phase upgrade commands, and the reassignment files for growing and shrinking a cluster. Host names, service names and sample output are illustrative; adapt the stop and start commands to how your brokers are run.
#!/usr/bin/env bashset -euo pipefailBS=broker1.example.com:9092,broker2.example.com:9092,broker3.example.com:9092BROKERS=(broker1.example.com broker2.example.com broker3.example.com)GATE_TIMEOUT_SECONDS=1800under_replicated() { # Prints one line per under-replicated partition. Empty output means none. bin/kafka-topics.sh --bootstrap-server "$BS" --describe --under-replicated-partitions}wait_for_full_isr() { local waited=0 while [ -n "$(under_replicated)" ]; do if [ "$waited" -ge "$GATE_TIMEOUT_SECONDS" ]; then echo "Gate not passed after ${GATE_TIMEOUT_SECONDS}s. Stopping here: a person must look." >&2 exit 1 fi sleep 15 waited=$((waited + 15)) done}# Pre-flight: refuse to start on a cluster that is not healthy.if [ -n "$(under_replicated)" ]; then echo "Under-replicated partitions exist. Not starting maintenance." >&2 exit 1fibin/kafka-metadata-quorum.sh --bootstrap-server "$BS" describe --statusif [ -n "$(bin/kafka-reassign-partitions.sh --bootstrap-server "$BS" --list | grep -v 'No partition reassignments')" ]; then echo "A partition reassignment is in progress. Not starting maintenance." >&2 exit 1fifor host in "${BROKERS[@]}"; do echo "== $host: controlled shutdown" # A normal stop (SIGTERM). The unit's stop timeout must be long enough for # controlled shutdown to finish, or the service manager will hard-kill it. ssh "$host" 'sudo systemctl stop kafka' # ... apply the change here: new package, new config, OS patch ... echo "== $host: start" ssh "$host" 'sudo systemctl start kafka' echo "== $host: waiting for every partition to be fully in sync" sleep 30 # give the broker time to register, so the first check is meaningful wait_for_full_isrdoneecho "All brokers restarted. Leadership is rebalanced by the controller, or run:"echo "bin/kafka-leader-election.sh --bootstrap-server $BS --election-type preferred --all-topic-partitions"Why it is written this way
- The gate asks the cluster whether any partition is under-replicated. It does not ask the restarted broker whether it is up.
- The gate has a timeout that stops the script. A broker that never catches up is a problem for a person; continuing would be the incident from the real-world section.
- The bootstrap list names several brokers so the check still works while one of them is down.
- The exact text printed by
--listwhen nothing is running may differ by version. Confirm it on your cluster before relying on the filter. - Controllers need the same treatment in a separate loop: one at a time, checking
kafka-metadata-quorum.sh describe --statusfor a leader and caught-up voters between them.
BS=broker1.example.com:9092# --- Before: what does the cluster use today? ---bin/kafka-features.sh --bootstrap-server $BS describe# Illustrative output on a 3.9 cluster:# Feature: kraft.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 0 Epoch: 5# Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 3.9-IV0 FinalizedVersionLevel: 3.9-IV0 Epoch: 5# --- Phase 1: new software, old feature level ---# Rolling restart of controllers, then brokers, with the new version installed# (see rolling-restart.sh). Afterwards the supported range has grown and the# finalized level has not moved:bin/kafka-features.sh --bootstrap-server $BS describe# Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 4.0-IV3 FinalizedVersionLevel: 3.9-IV0 Epoch: 9# Rollback at this point = rolling restart with the old version.# Now wait. Watch error rates, request latency, consumer lag through at least# one full traffic cycle.# --- Phase 2: finalise (a separate change, days later) ---bin/kafka-features.sh --bootstrap-server $BS upgrade --release-version 4.0bin/kafka-features.sh --bootstrap-server $BS describe# Feature: metadata.version SupportedMinVersion: 3.3-IV3 SupportedMaxVersion: 4.0-IV3 FinalizedVersionLevel: 4.0-IV3 Epoch: 12Why it is written this way
SupportedMaxVersiondescribes the software;FinalizedVersionLeveldescribes the cluster. Between the phases they differ, and that difference is what keeps rollback possible.- Use the release version you actually installed in place of 4.0. The command and the two-phase shape are the same for every 4.x release.
- The sample output follows the format shown in the Kafka documentation; the version strings and epochs here are illustrative.
- Read the upgrade notes for every release between your old and new version before phase 2. They say whether a metadata downgrade is supported and which behaviour finalisation switches on.
BS=broker1.example.com:9092# Brokers 4 and 5 are new, running and empty. Spread two topics over all five.cat > topics-to-move.json <<'JSON'{"version":1,"topics":[{"topic":"shipments.events"},{"topic":"tracking.updates"}]}JSON# 1. Propose a plan. Nothing moves yet. The output contains the CURRENT assignment# (save it as rollback.json) and the PROPOSED one (save it as expand.json).bin/kafka-reassign-partitions.sh --bootstrap-server $BS \ --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3,4,5" --generate# 2. Start it, limited to 40 MB/s between brokers.bin/kafka-reassign-partitions.sh --bootstrap-server $BS \ --reassignment-json-file expand.json --execute --throttle 40000000# 3. Check until every partition is completed. On completion this removes the throttle.bin/kafka-reassign-partitions.sh --bootstrap-server $BS \ --reassignment-json-file expand.json --verify# Illustrative output when finished:# Status of partition reassignment:# Reassignment of partition [shipments.events,0] is completed# ...# Clearing broker-level throttles on brokers 1,2,3,4,5# Clearing topic-level throttles on topics shipments.events,tracking.updates# 4. Spread leadership according to the new replica lists.bin/kafka-leader-election.sh --bootstrap-server $BS \ --election-type preferred --all-topic-partitionsWhy it is written this way
--generatespreads the named topics evenly by count. It knows nothing about partition size or traffic, so review the proposal, and move the heaviest topics in their own batches.- Saving the current assignment is the rollback: it can be fed to
--executeto move everything back. - The throttle must exceed the write rate of the partitions being moved. Watch that the new replicas' lag falls; if it does not, raise the limit by re-running
--executewith--additionaland a higher--throttle. - Until
--verifyreports completion and clears the throttles, ordinary replication for those topics is rate-limited as well.
BS=broker1.example.com:9092# 1. Cordon broker 6 so no new replicas are placed on it (per the 4.4 documentation).bin/kafka-configs.sh --bootstrap-server $BS --alter \ --add-config cordoned.log.dirs="*" --entity-type brokers --entity-name 6# 2. Move every replica off broker 6. The tool cannot generate this plan:# build a JSON that lists, for each partition with a replica on 6,# the same replicas with 6 replaced by another broker.bin/kafka-reassign-partitions.sh --bootstrap-server $BS \ --reassignment-json-file drain-broker-6.json --execute --throttle 40000000bin/kafka-reassign-partitions.sh --bootstrap-server $BS \ --reassignment-json-file drain-broker-6.json --verify# 3. Confirm broker 6 holds nothing (JSON output lists replicas per log directory).bin/kafka-log-dirs.sh --bootstrap-server $BS --describe --broker-list 6# 4. Stop the broker, then remove its registration from the cluster metadata.ssh broker6.example.com 'sudo systemctl stop kafka'bin/kafka-cluster.sh unregister --bootstrap-server $BS --id 6Why it is written this way
- The broker is stopped only after it holds no replicas, so its shutdown changes no ISR.
- When writing the plan, keep replicas of one partition in different racks or zones, and check the remaining brokers have room for the data they receive.
- Unregistering matters: a broker that is merely stopped stays in the metadata as a fenced broker and keeps appearing in tools and metrics.
- If your version does not have
cordoned.log.dirs, skip step 1 and avoid creating topics or running other reassignments until the drain is complete.
Configuration
Settings that govern planned operations. Defaults are from the Apache Kafka 4.4 configuration reference.
controlled.shutdown.enableDefault true
- What it does
- On a normal stop the broker has the controller move its leaderships to other replicas before it exits.
- When to change it
- Leave on. Make sure whatever stops the broker (systemd, Kubernetes) waits long enough before a hard kill; on brokers with many partitions that is minutes, not seconds.
auto.leader.rebalance.enableDefault true
- What it does
- The controller periodically returns leadership to preferred replicas that are in sync.
- When to change it
- Leave on so that a rolling restart does not end with all leaders on the last brokers to be restarted. If off, run a preferred leader election at the end of every roll.
leader.imbalance.check.interval.secondsDefault 300
- What it does
- How often the controller checks for partitions not led by their preferred replica.
- When to change it
- Explains the delay between a broker returning and it leading partitions again. Rarely changed.
min.insync.replicasDefault 1
- What it does
- The smallest ISR size at which
acks=allwrites are accepted. - When to change it
- With replication factor 3, set 2: one broker can be down for maintenance with writes still accepted and still on two disks. With min ISR equal to the replication factor, every restart refuses writes.
leader.replication.throttled.rate / follower.replication.throttled.rateDefault 9223372036854775807 (no limit)
- What it does
- Upper bound in bytes per second on throttled replication traffic on the leader side and the follower side. Dynamic broker settings, normally set for you by
kafka-reassign-partitions.sh --throttle. - When to change it
- Do not set by hand unless you have to. After any reassignment, confirm they are gone with
kafka-configs.sh --describe --entity-type brokers. The documentation suggests keeping any limit above 1 MB/s.
leader.replication.throttled.replicas / follower.replication.throttled.replicasDefault "" (none)
- What it does
- Topic settings listing which replicas the throttle rates apply to. Also set and cleared by the reassignment tool.
- When to change it
- A leftover value here together with a leftover rate is what makes ordinary replication slow weeks after a reassignment.
cordoned.log.dirsDefault "" (none)
- What it does
- Log directories on a broker that should receive no new replicas.
*cordons the whole broker. - When to change it
- Set per broker before draining it for decommissioning. Remove it if the broker is to stay in service.
broker.rackDefault null
- What it does
- The rack or availability zone of a broker, used to spread a partition's replicas when they are placed.
- When to change it
- Set it on every broker, including new ones, before reassigning. A new broker without it can receive replicas that put two copies in one zone.
metadata.max.age.ms (clients)Default 300000 (5 minutes)
- What it does
- How often a client refreshes metadata even without an error, which is how it discovers new partitions and brokers.
- When to change it
- Sets the worst-case delay before producers use added partitions. Usually left alone; plan partition increases around it.
auto.offset.reset (consumer)Default latest
- What it does
- Where a group starts on a partition for which it has no committed offset.
- When to change it
- Decides whether records written to a newly added partition before the consumer finds it are read or skipped. For topics whose partition count may grow, do not leave this at
latestwithout thinking about that gap.
Hands-on exercise
Troubleshoot: the restart script that refused orders
Work out from a script log and client symptoms why a routine rolling restart refused writes, reproduce the failure in the cluster explorer, then perform the restart correctly.
Steps · 0 of 7 done
Rolling restart of three brokers
Simulation · not a real clusterorders.events with 3 partitions, replication factor 3, min.insync.replicas 2. First reproduce the failed restart, then run a correct one: stop, wait for the ISR, restart, next.
What you should see
The script continued when broker 1's port opened. At that moment broker 1 was running but had not caught up, so it was in no ISR; stopping broker 2 left every partition with one in-sync replica. In the widget, with brokers 1 and 2 both out, all three partitions show ISR 1, Produce is rejected on all three with NotEnoughReplicasException, and Offline partitions stays 0 because broker 3 still leads. In the correct run, each stop makes Under-replicated 3 while Produce still succeeds, and each restart brings it back to 0 once the broker has caught up. No write is refused at any point. After the three restarts, broker 1 leads two partitions and broker 3 leads none, because a returning broker comes back as a follower; the preferred leader election puts leadership back to one partition per broker. The fix for the script is to wait for under-replicated partitions to reach 0, not for a port.
What can go wrong
Gating on a port, a process check or a sleep
- Symptom
- A rolling restart that has worked for months suddenly causes
NotEnoughReplicasExceptionfor several minutes, usually on the day one broker takes longer than usual to come back. - Cause
- The script moved to the next broker while the previous one was still recovering logs or catching up, so two replicas of many partitions were out of sync together.
- Fix
- Gate on
kafka-topics.sh --describe --under-replicated-partitionsreturning nothing, with a timeout that stops the run and alerts a person.
Hard-killing brokers during a roll
- Symptom
- Each restart causes several seconds of producer timeouts instead of none, and brokers take much longer to return.
- Cause
- The stop timeout of the service manager is shorter than a controlled shutdown, so the process is killed. Leaders move only after the session timeout and the broker must recover its logs at startup.
- Fix
- Raise the stop timeout (systemd
TimeoutStopSec, KubernetesterminationGracePeriodSeconds) well beyond the longest observed shutdown, and confirm clean shutdown in the broker log.
Finalising in the same change as the software rollout
- Symptom
- A regression is found after an upgrade and the team discovers it cannot return to the previous version.
- Cause
kafka-features.sh upgradewas run as soon as the last broker restarted. The cluster now uses a metadata version the old software may not read, and downgrade between those versions is not supported.- Fix
- Make finalisation a separate, later change. Before it, read the upgrade notes for the exact versions involved.
Adding partitions to a keyed topic that is in use
- Symptom
- After a partition increase, a consumer applies events for the same order or account out of order, or per-key state stores look inconsistent.
- Cause
- The key-to-partition mapping changed with the partition count. New records for a key go to a different partition from its older, possibly unprocessed, records.
- Fix
- Size keyed topics for growth at creation. If more partitions are needed later, migrate to a new topic, or pause producers and let consumers drain the old partitions before the change.
Reassigning without a throttle, or leaving one behind
- Symptom
- Latency and lag rise cluster-wide during an expansion; or followers lag persistently some time after one.
- Cause
- Moving partitions copies their full contents and competes with client traffic. A throttle that is never cleared limits normal replication afterwards.
- Fix
- Always use
--throttle, move in batches, and finish every reassignment with--verify.
Expecting a new broker to take load by itself
- Symptom
- Two brokers are added to an overloaded cluster and nothing improves; the new brokers sit idle.
- Cause
- Kafka assigns existing partitions to new brokers only when told to. Only topics created afterwards may place replicas there.
- Fix
- Follow the addition with a planned, throttled reassignment. Add capacity before the cluster is too busy to afford the copy.
How to troubleshoot
During a rolling restart, the gate never passes: under-replicated partitions stay above 0
Check
`kafka-topics.sh --bootstrap-server … --describe --under-replicated-partitions`; on the restarted broker `remainingLogsToRecover` and `kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica`
- How to read it
- If only the restarted broker is missing from the ISRs and its lag is falling, wait. Logs still being recovered mean the previous stop was not clean. If a different broker is missing, something else is wrong: stop the roll.
Producers get NotEnoughReplicasException during maintenance
Check
`kafka-topics.sh --bootstrap-server … --describe --under-min-isr-partitions`
- How to read it
- Two replicas of those partitions are out of sync at once: the broker being restarted plus one that had not finished catching up, or one that was already unhealthy. Stop the roll and let the missing replicas rejoin.
Stopping a broker takes very long, or it is killed by the service manager
Check
The broker's server log during shutdown for controlled shutdown progress; the service unit's stop timeout; topics with replication factor 1 via `kafka-topics.sh --describe`
- How to read it
- Controlled shutdown waits for leaderships to move, and cannot complete for partitions with no other live replica. Raise the timeout, and find single-replica topics before the maintenance.
After the roll, a few brokers handle most of the traffic
Check
`kafka.server:type=ReplicaManager,name=LeaderCount` per broker; `PreferredReplicaImbalanceCount` on the controller
- How to read it
- Leadership stayed where it moved during the restarts. It corrects itself within the controller's check interval if automatic rebalancing is on; otherwise run
kafka-leader-election.sh --election-type preferred --all-topic-partitions.
Unsure whether an upgrade has been finalised, or whether the cluster can be rolled back
Check
`kafka-features.sh --bootstrap-server … describe`, comparing `FinalizedVersionLevel` with `SupportedMaxVersion` for `metadata.version`; the `CurrentMetadataVersion` metric
- How to read it
- Finalised level below the supported maximum: phase 1 only, rollback by reinstalling the old version is open. Equal: finalised, and the upgrade notes decide whether a downgrade exists.
A reassignment has been running for hours without finishing
Check
`kafka-reassign-partitions.sh --bootstrap-server … --reassignment-json-file plan.json --verify`; `ReassignmentBytesInPerSec` on the destination; `kafka.server:type=FetcherLagMetrics,name=ConsumerLag` for the moving replicas; `BytesInPerSec` for the topic
- How to read it
- Lag falling: it is progressing. Lag flat or rising while reassignment bytes sit at the throttle: the throttle is below the topic's write rate. Raise it with
--execute --additional --throttle.
After adding partitions, a consumer group seems to have missed records
Check
`kafka-consumer-groups.sh --bootstrap-server … --describe --group …` for the new partitions' current offset and log end offset; the group's `auto.offset.reset`
- How to read it
- If the group's first committed offset on a new partition is above the partition's start, it began at the end and skipped what was produced before it discovered the partition. Those records are still in the log and can be reprocessed by resetting the group's offset for that partition.
Key takeaways
- Every planned change is a rolling restart: one broker out, changed, back, fully in sync, then the next.
- The gate between brokers is zero under-replicated partitions, asked of the cluster. A port, a process check or a timer is not a gate.
- Always stop brokers with a controlled shutdown and give it time to finish.
- An upgrade has two phases. New software can be rolled back; finalising
metadata.versionshould be treated as one-way and done later, as its own change. - 4.x is KRaft only. KRaft clusters at 3.3 or later upgrade directly; ZooKeeper clusters must migrate to KRaft on a 3.x bridge release first.
- New brokers are empty until you reassign partitions to them, with a throttle, and finish with
--verify. - Adding partitions changes where keys go and cannot be undone. For keyed, ordered data, plan a migration instead.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
You are rolling a 6-broker cluster (replication factor 3, min.insync.replicas=2). Broker 3 was restarted 4 minutes ago. Its process is up, its port accepts connections, and it appears in the controller's active broker count. kafka-topics.sh --describe --under-replicated-partitions still prints 212 lines, all with broker 3 missing from Isr, and that number was 480 two minutes ago.
What do you do next?
Scenario 2 of 3
All brokers and controllers of a cluster were upgraded from 3.9 to 4.0 software this afternoon. kafka-features.sh describe shows metadata.version FinalizedVersionLevel 3.9-IV0 and SupportedMaxVersion 4.0-IV3. A colleague proposes finalising now 'to complete the change', before the overnight batch peak has run on the new version.
What is the better plan?
Scenario 3 of 3
Topic account.events has 12 partitions, keyed by account id, and its consumer applies events in order to build account balances. Lag is growing at peak because 12 consumers are not enough. Someone suggests kafka-topics.sh --alter --partitions 24 right now, with producers running and about 300,000 unprocessed records in the topic.
What is the main risk of doing that?
References
References: Apache Kafka documentation: Upgrading (rolling upgrade, finalisation, downgrade notes, 4.0 changes) ↗ · Apache Kafka documentation: Basic Kafka operations (graceful shutdown, modifying topics, expanding, decommissioning, throttling) ↗ · Apache Kafka documentation: Compatibility (client and broker matrix for 4.0) ↗ · Apache Kafka documentation: Operations, KRaft (feature versions, controller membership, ZooKeeper migration note) ↗ · Apache Kafka documentation: KRaft vs ZooKeeper (removed configurations and metrics) ↗ · Apache Kafka documentation: Broker configuration reference ↗ · Apache Kafka documentation: Java version ↗ · KIP-896: Remove old client protocol API versions in Kafka 4.0 ↗ · KIP-584: Versioning scheme for features ↗ · Report a problem