Skip to content

Path 10 · lesson 3 of 6

Advanced28 minApache Kafka 4.x, KRaft only (checked against the 4.4 Upgrading, Compatibility, Basic Operations and KRaft pages) · Java 21 · Spring Boot 3.5 with Spring for Apache Kafka 3.3 and kafka-clients 3.9

Rolling upgrades and scaling a cluster

How to restart, upgrade, grow and shrink a Kafka cluster one broker at a time without refusing a write, and which steps cannot be undone.

What you will learn

  • Restart every broker in a cluster in turn while keeping every partition writable with acks=all
  • Name the check that must pass between two brokers and explain why a port or process check is not it
  • Separate the two phases of an upgrade, new software and feature finalisation, and say which one can be rolled back
  • Plan an upgrade path from a 3.x cluster to 4.x, including clusters still on ZooKeeper
  • Add a broker and move partitions onto it under a throttle, and remove a broker cleanly
  • Judge when increasing a topic's partition count is safe and what it breaks

Before you start

The concept

Almost every planned change to a Kafka cluster is the same operation repeated: take one broker out, change it, put it back, wait until it is fully in sync, move to the next. A configuration change, an operating system patch, a new Kafka version and a hardware replacement all use it. It works because of replication: with replication factor 3 and min.insync.replicas=2, one broker may be absent at any moment and every partition still has two in-sync replicas.

That sentence also states the limit. The procedure is safe only while at most one replica of any partition is out of sync. A rolling restart is a sequence of deliberate single-broker failures, and the previous lesson showed what the second concurrent failure does. So the whole skill is in the waiting: knowing what to wait for and refusing to continue until it is true.

OperationHow it is doneWhat makes it safeReversible?
Restart or reconfigure brokersRolling restart, one broker at a timeNo under-replicated partitions before each stop; controlled shutdownYes
Upgrade the Kafka softwareRolling restart with the new versionSame, and the cluster still speaks the old metadata versionYes, by rolling back to the old software
Finalise the upgradekafka-features.sh upgrade --release-version …Done only after the new software has run long enough to trustOften not: depends on the metadata versions involved
Add a brokerStart it, then reassign partitions to itThrottle on the data movementYes, by reassigning back
Remove a brokerReassign everything away, stop it, unregister itIt holds no replicas when it stopsYes, until it is unregistered and wiped
Increase a topic's partitionskafka-topics.sh --alter --partitionsNothing makes it safe for keyed data in flightNo: partitions cannot be reduced

One at a time

Simulation · not a real cluster

Stop one broker and watch the Under-replicated counter. Restart it and wait for the counter to return to 0 before stopping the next. Then try stopping two at once and press Produce.

Why it exists

A cluster that could only be changed by stopping it would force a choice between running old software and scheduling outages. Kafka is the shared dependency of many services, so an outage window for Kafka is an outage window for all of them. Rolling operations exist so that the cluster's own maintenance is invisible to its clients: a producer sees a leader move and retries, a consumer sees the same and continues.

Upgrades add a second problem. During a rolling upgrade, brokers of two versions run side by side and must understand each other, and what they write to disk and to the metadata log must be readable by both. Kafka solves this by separating the software version from the feature level the cluster uses. New software starts out behaving like the old version. Only when every node runs the new software do you switch the cluster to the new feature level, as an explicit, separate step.

Phase 1: roll out the software

  • Each broker is restarted with the new version
  • The cluster keeps using the old metadata.version
  • Mixed versions run together safely
  • Rollback is another rolling restart with the old version

Phase 2: finalise

  • One command raises metadata.version and other features for the whole cluster
  • New record formats and protocols come into use
  • Old software may no longer be able to read the metadata
  • Rollback is possible only if no metadata changes lie between the versions

Scaling has its own reason for care. Kafka never moves data by itself. A new broker receives nothing until you assign partitions to it, and moving a partition means copying all of it. On a busy cluster that copy competes with the traffic you are scaling for, which is why capacity should be added before it is urgent. The Break card One server too many is a sourced example, from a different streaming system, of a routine capacity addition that pushed a whole fleet over a limit.

How it works inside

The rolling restart procedure

  1. Pre-flight. The cluster must be healthy before you start: no under-replicated partitions, no offline partitions, an active controller, no reassignment running. If any of these fail, the maintenance does not begin.
  2. Stop one broker with a controlled shutdown. A normal stop signal, never a hard kill. The broker asks the controller to move its leaderships to other in-sync replicas first, then flushes its logs and exits. Clients see leader changes of a few milliseconds per partition and retry.
  3. Apply the change and start the broker. It registers, catches up on metadata, becomes a follower for all its partitions and fetches what it missed.
  4. Wait for the gate. Do not continue until under-replicated partitions are back to 0. That is the proof that the broker is in every ISR again.
  5. Repeat for the next broker. Many teams restart the broker hosting the fewest leaders first and take a short pause after each to watch client error rates.
  6. Finish. Leadership ends up concentrated on the brokers restarted earliest. The controller's automatic preferred leader election corrects this within its check interval, or you can run kafka-leader-election.sh --election-type preferred --all-topic-partitions.

Controlled shutdown has a precondition stated in the documentation: it succeeds only if every partition on the broker has another live replica to take over. A topic with replication factor 1 has none, so its partitions are simply unavailable while its broker is down. Rolling operations assume replication factor 3 and min.insync.replicas=2 for everything that matters; with replication factor 2 and min.insync.replicas=2, every restart refuses writes.

Controllers are rolled the same way, one at a time, keeping a majority of the quorum running throughout. With three controllers that means never having two down together. A common practice is to restart the standby controllers first and the active one last, so that there is one controller election and not several. If a server runs in combined mode (process.roles=broker,controller) each restart takes out a broker and a voter at once, one reason the documentation advises against combined mode for critical deployments.

Software version, metadata.version and finalisation

In KRaft the cluster's feature level is itself a piece of replicated metadata. metadata.version says which record formats and inter-broker behaviour the cluster uses. It is finalised: stored in the metadata log, the same for every node, and changed only by an explicit command. It replaces the inter.broker.protocol.version setting that ZooKeeper-based clusters edited in each broker's properties file. Other features are versioned the same way, for example kraft.version and eligible.leader.replicas.version. kafka-features.sh describe shows, for each feature, the range the software supports and the level currently finalised.

The documented upgrade procedure is short. Upgrade the brokers one at a time: shut down, update the code, restart. Once all are on the new version, verify that the cluster's behaviour and performance meet expectations. Then finalise with bin/kafka-features.sh --bootstrap-server … upgrade --release-version <version>.

Finalising can also switch on behaviour, which is another reason to give it its own change window. The 4.0 notes give two examples: the new consumer group protocol (KIP-848) becomes available on the server when the upgrade to 4.0 is finalised, and so does the strengthened transaction protocol.

From 3.x to 4.x

Kafka 4.0 removed ZooKeeper mode. A 4.x broker cannot start against ZooKeeper, so there is no rolling upgrade from a ZooKeeper-based cluster straight to 4.x. The upgrade notes give the supported paths:

Your cluster todayPath to 4.x
KRaft mode, software and metadata.version at 3.3 or laterRolling upgrade directly to 4.x, then finalise.
KRaft mode, older than 3.3Upgrade to 3.9 first, then to 4.x.
ZooKeeper modeUpgrade to a bridge release (3.9 is the last one), migrate the cluster from ZooKeeper to KRaft there following the 3.9 documentation, and only then upgrade to 4.x.
From the Upgrading page of the Apache Kafka 4.4 documentation. The same note is repeated for each 4.x release.

Two more things changed at 4.0 that an upgrade plan must check. The broker needs Java 17 or later (clients still run on Java 11). And settings, tools and metrics that existed only for ZooKeeper are gone: zookeeper.connect, the --zookeeper flag, inter.broker.protocol.version, log.message.format.version and several controller metrics. Configuration files and dashboards carried over from 3.x need a review before the first 4.x broker starts.

Client and broker compatibility

Kafka clients and brokers negotiate protocol versions per request, so a client does not have to match the broker version. Clients and brokers are upgraded independently and in either order, within limits. Kafka 4.0 removed the oldest protocol versions (KIP-896): brokers on 4.x require clients at version 2.1 or later, and 4.x Java clients require brokers at 2.1 or later. The compatibility page lists 3.x clients as fully compatible with 4.0 brokers.

In practice: a Spring Boot 3.5 service, which uses kafka-clients 3.9, works against a 4.x cluster without changes. What it cannot do is use features that need a newer client. Before upgrading brokers to 4.x, find every client older than 2.1, and check clients that are not the Apache Java client (librdkafka-based and others) against their own compatibility notes, since KIP-896 affects them through the protocol versions they send.

Adding brokers

A new broker gets a unique node.id, has its storage formatted with the cluster's id, and is started. It joins, and then does nothing, because no partition is assigned to it. Only topics created afterwards may place replicas there. To use it, you reassign existing partitions with kafka-reassign-partitions.sh: --generate proposes an even spread of the topics you name over the broker list you give, --execute starts it, --verify reports completion. The tool does not look at partition sizes or traffic, so which topics to move is your decision.

For each moved partition the new broker is added as a follower, copies the whole partition, joins the ISR, and then one old replica is deleted. Always pass --throttle. The limit must be above the write rate of the partitions being moved, or the new replica never catches up, and --verify must be run at the end because it removes the throttle. Move in batches (a few topics at a time): a smaller plan finishes sooner, is easier to cancel, and keeps the number of partitions with an extra replica small.

Removing a broker

Decommissioning is the reverse, with one extra step at each end. The 4.4 documentation describes it as: cordon the broker so that no new replicas are placed on it (cordoned.log.dirs set to * for that broker; a recent setting, so check that your version has it), reassign every replica it holds to other brokers, shut it down once it holds none, and unregister it with kafka-cluster.sh unregister --id … so that the controller forgets it. The tool cannot generate a decommissioning plan for you. Before starting, check that the remaining brokers have the disk and network capacity for what they are about to receive, and that enough brokers remain for the replication factor.

Increasing partitions

kafka-topics.sh --alter --topic … --partitions N adds partitions at once. It is the cheapest scaling command to type and the one with the most lasting effects, which the documentation lists:

  • Keys move. The default partitioner chooses a partition from the hash of the key and the partition count. With a new count, the same key can map to a different partition. Existing records stay where they are, so records for one key now sit in two partitions and their relative order is no longer guaranteed across that boundary.
  • Kafka does not redistribute existing data. Old partitions keep their backlog; new ones start empty.
  • It cannot be undone. Kafka does not support reducing the number of partitions.
  • Consumers can miss the first records. A group with no committed offset for the new partition starts where auto.offset.reset says. With latest, records produced before the consumer discovers the partition are skipped. Recent client versions accept by_duration:<duration> for this case; with older clients, earliest avoids the gap at the cost of nothing, since the partition is new.
  • Clients notice late. Producers and classic-protocol consumers learn of new partitions on a metadata refresh (metadata.max.age.ms, 5 minutes by default).
  • Never on internal topics. Changing the partition count of __consumer_offsets, __transaction_state or other internal topics breaks the coordinator mapping.

So it is safe for topics without keys, and for keyed topics only when nobody depends on per-key order across the change or the topic can be drained first. For a keyed topic that needs more partitions and strict ordering, the usual route is a new topic with the target count and a controlled migration. Choosing a partition count with room to grow is covered in Partition sizing and parallelism.

Rolling out configuration changes

Broker settings differ in how they change. The configuration reference gives each an update mode: read-only settings need a broker restart, so a rolling restart; per-broker and cluster-wide settings can be changed on a running cluster with kafka-configs.sh. Topic settings such as retention are always dynamic. Prefer the dynamic route where it exists, and change one broker or one topic first.

Client settings change by redeploying the application, and for consumers every restarted instance leaves and rejoins its group. Roll out one instance first and compare its metrics with the others. Restart consumers a few at a time, and use static membership or the cooperative assignor so that a rolling deployment does not become a series of full rebalances; see Consumer rebalancing. Changes that alter what is written (compression type, a serializer, a partitioner or the key) deserve the most care, because every consumer of the topic will read the result.

Architecture

Drawing diagram…

A flowchart. A pre-flight check leads to a decision: if the cluster is not healthy, stop. Otherwise one broker is shut down in a controlled way, upgraded and started. A loop waits until under-replicated partitions are back to zero, then repeats for the next broker. When all nodes run the new software with the old metadata version, the cluster is observed for days. If a problem is found, roll back with the old version. If not, finalise with the features tool, after which the new feature level is in use and the step is treated as one-way.

flowchart TD
  A["Pre-flight: under-replicated is 0, controller active, no reassignment"] --> B{"Healthy?"}
  B -->|"no"| X["Stop. Fix the cluster first"]
  B -->|"yes"| C["Controlled shutdown of one broker"]
  C --> D["Install new version, start broker"]
  D --> E{"Under-replicated back to 0?"}
  E -->|"not yet"| W["Wait and watch catch-up"]
  W --> E
  E -->|"yes"| F{"More brokers?"}
  F -->|"yes"| C
  F -->|"no"| G["All nodes on new software, old metadata.version"]
  G --> H["Observe for days: errors, latency, lag"]
  H --> I{"Problem found?"}
  I -->|"yes"| R["Roll back: rolling restart with old version"]
  I -->|"no"| J["Finalise: kafka-features upgrade"]
  J --> K["New feature level in use. Treat as one-way"]
A rolling upgrade as two phases with a gate between brokers and an observation period before finalising.
  • The loop in the middle is the rolling restart. Its only exit is the under-replicated gate, which is why a slow catch-up makes the upgrade longer and not less safe.
  • Everything above the finalise box can be undone by repeating the loop with the old binaries, because the cluster has not yet written anything the old version cannot read.
  • The observation period is drawn as its own step on purpose. Problems that matter (a latency regression at peak, a client that misbehaves against the new brokers) often need a full traffic cycle to show.
  • Finalising is a single command against the whole cluster, with no rolling and no gate. All the care goes into deciding when to run it.

Message flow

Drawing diagram…

A sequence diagram. The operator script confirms there are no under-replicated partitions, then stops broker 1. Broker 1 requests a controlled shutdown, the controller moves its leaderships to in-sync replicas such as broker 2, and tells broker 1 it may exit. The producer refreshes metadata and writes to the new leader with two in-sync replicas. The script starts broker 1 with the new version, it registers, fetches the missed records as a follower and is added back to the ISR. The script checks again, finds no under-replicated partitions, and continues with broker 2.

sequenceDiagram
  participant O as Operator script
  participant B1 as Broker 1
  participant C as Active controller
  participant B2 as Broker 2
  participant P as Producer
  O->>B2: Describe under-replicated partitions
  B2-->>O: None
  O->>B1: Stop with a normal signal
  B1->>C: Request controlled shutdown
  C->>C: Move leaderships of broker 1 to in-sync replicas
  C-->>B2: You lead orders.events-0
  C-->>B1: Safe to shut down
  B1->>B1: Flush logs and exit
  P->>B2: Metadata refresh then produce to the new leader
  B2-->>P: Acknowledged, ISR is 2 and 3
  O->>B1: Start with the new version
  B1->>C: Register and catch up on metadata
  B1->>B2: Fetch missed records as follower
  B2->>C: Propose ISR 2 and 3 and 1
  O->>B2: Describe under-replicated partitions
  B2-->>O: None
  Note over O: Gate passed, continue with broker 2
One step of a rolling restart: check, controlled shutdown, restart, catch-up, check again.
  1. The same check opens and closes the step. It is asked of the cluster, not of the broker being restarted, because a broker cannot judge whether it is in sync.
  2. Leadership moves before the process exits, so producers are redirected by a metadata refresh and do not wait for a session timeout as they would after a crash.
  3. While broker 1 is down, writes with acks=all succeed with an ISR of two. That is the margin being spent, and why nothing else may be down at the same time.
  4. The broker's return to the ISR is proposed by the partition leaders once it has caught up. How long that takes depends on how much was written while it was away.

In a real system

A typical incident during an upgrade from Kafka 3.9 to 4.x at a logistics company. Six brokers, three dedicated controllers, already on KRaft. Topic shipments.events has 48 partitions, replication factor 3 and min.insync.replicas=2, keyed by shipment id. The upgrade is automated with a script that had been used for configuration restarts for a year.

TimeEventWhat the tools showed (illustrative)The flaw
21:00Script starts. For each broker: stop, install, start, wait until port 9092 accepts connections, sleep 60 seconds, next.Under-replicated partitions 0 at the start.The gate is a port check and a timer.
21:04Broker 1 back. It had been down 3 minutes and catches up in 20 seconds.Under-replicated back to 0 well inside the 60 seconds.The timer happened to be long enough, as it had been for a year.
21:09Broker 2 is stopped by a hard kill: the service unit's stop timeout was shorter than the controlled shutdown needed.Server log on restart shows log recovery running for every partition.No controlled shutdown, so leaders moved only after the session timeout and recovery was needed.
21:11Broker 2's port opens. It is still recovering logs and has not begun to fetch. The 60 seconds pass.Under-replicated 410 and not falling.Running is not in sync.
21:12Script stops broker 3.Under-min-ISR 137. Producers of shipments.events log NotEnoughReplicasException.Two replicas of 137 partitions are out at once.
21:26Broker 3 is back and in sync; broker 2 finishes recovery a few minutes later.Under-min-ISR 0, then under-replicated 0.Fourteen minutes of refused writes, caused entirely by the procedure.
21:40The script completes and, as its last line, runs kafka-features.sh upgrade.Finalised metadata.version at the new level.No observation period.
Next dayA consumer latency regression appears at peak. The team wants to roll back to 3.9 while investigating.The upgrade notes for 4.0 state that metadata downgrade is not supported.Rollback was given away twelve hours earlier for no benefit.
Illustrative timeline of a typical incident. Every step had worked many times before.

Three changes came out of the review. The gate between brokers became the cluster's own answer to whether any partition is under-replicated, with a timeout that stops the script and pages a person. The service unit's stop timeout was raised so that controlled shutdown can finish, and the script checks the broker's log for a clean shutdown. And finalisation was removed from the script: it is now a separate change, made by a person, no sooner than a week after the last broker is upgraded.

Java and Spring Boot

A restart script with the right gate, the two-phase upgrade commands, and the reassignment files for growing and shrinking a cluster. Host names, service names and sample output are illustrative; adapt the stop and start commands to how your brokers are run.

ops/rolling-restart.shShell
#!/usr/bin/env bashset -euo pipefailBS=broker1.example.com:9092,broker2.example.com:9092,broker3.example.com:9092BROKERS=(broker1.example.com broker2.example.com broker3.example.com)GATE_TIMEOUT_SECONDS=1800under_replicated() {  # Prints one line per under-replicated partition. Empty output means none.  bin/kafka-topics.sh --bootstrap-server "$BS" --describe --under-replicated-partitions}wait_for_full_isr() {  local waited=0  while [ -n "$(under_replicated)" ]; do    if [ "$waited" -ge "$GATE_TIMEOUT_SECONDS" ]; then      echo "Gate not passed after ${GATE_TIMEOUT_SECONDS}s. Stopping here: a person must look." >&2      exit 1    fi    sleep 15    waited=$((waited + 15))  done}# Pre-flight: refuse to start on a cluster that is not healthy.if [ -n "$(under_replicated)" ]; then  echo "Under-replicated partitions exist. Not starting maintenance." >&2  exit 1fibin/kafka-metadata-quorum.sh --bootstrap-server "$BS" describe --statusif [ -n "$(bin/kafka-reassign-partitions.sh --bootstrap-server "$BS" --list | grep -v 'No partition reassignments')" ]; then  echo "A partition reassignment is in progress. Not starting maintenance." >&2  exit 1fifor host in "${BROKERS[@]}"; do  echo "== $host: controlled shutdown"  # A normal stop (SIGTERM). The unit's stop timeout must be long enough for  # controlled shutdown to finish, or the service manager will hard-kill it.  ssh "$host" 'sudo systemctl stop kafka'  # ... apply the change here: new package, new config, OS patch ...  echo "== $host: start"  ssh "$host" 'sudo systemctl start kafka'  echo "== $host: waiting for every partition to be fully in sync"  sleep 30   # give the broker time to register, so the first check is meaningful  wait_for_full_isrdoneecho "All brokers restarted. Leadership is rebalanced by the controller, or run:"echo "bin/kafka-leader-election.sh --bootstrap-server $BS --election-type preferred --all-topic-partitions"

Why it is written this way

  • The gate asks the cluster whether any partition is under-replicated. It does not ask the restarted broker whether it is up.
  • The gate has a timeout that stops the script. A broker that never catches up is a problem for a person; continuing would be the incident from the real-world section.
  • The bootstrap list names several brokers so the check still works while one of them is down.
  • The exact text printed by --list when nothing is running may differ by version. Confirm it on your cluster before relying on the filter.
  • Controllers need the same treatment in a separate loop: one at a time, checking kafka-metadata-quorum.sh describe --status for a leader and caught-up voters between them.

Configuration

Settings that govern planned operations. Defaults are from the Apache Kafka 4.4 configuration reference.

controlled.shutdown.enableDefault true
What it does
On a normal stop the broker has the controller move its leaderships to other replicas before it exits.
When to change it
Leave on. Make sure whatever stops the broker (systemd, Kubernetes) waits long enough before a hard kill; on brokers with many partitions that is minutes, not seconds.
auto.leader.rebalance.enableDefault true
What it does
The controller periodically returns leadership to preferred replicas that are in sync.
When to change it
Leave on so that a rolling restart does not end with all leaders on the last brokers to be restarted. If off, run a preferred leader election at the end of every roll.
leader.imbalance.check.interval.secondsDefault 300
What it does
How often the controller checks for partitions not led by their preferred replica.
When to change it
Explains the delay between a broker returning and it leading partitions again. Rarely changed.
min.insync.replicasDefault 1
What it does
The smallest ISR size at which acks=all writes are accepted.
When to change it
With replication factor 3, set 2: one broker can be down for maintenance with writes still accepted and still on two disks. With min ISR equal to the replication factor, every restart refuses writes.
leader.replication.throttled.rate / follower.replication.throttled.rateDefault 9223372036854775807 (no limit)
What it does
Upper bound in bytes per second on throttled replication traffic on the leader side and the follower side. Dynamic broker settings, normally set for you by kafka-reassign-partitions.sh --throttle.
When to change it
Do not set by hand unless you have to. After any reassignment, confirm they are gone with kafka-configs.sh --describe --entity-type brokers. The documentation suggests keeping any limit above 1 MB/s.
leader.replication.throttled.replicas / follower.replication.throttled.replicasDefault "" (none)
What it does
Topic settings listing which replicas the throttle rates apply to. Also set and cleared by the reassignment tool.
When to change it
A leftover value here together with a leftover rate is what makes ordinary replication slow weeks after a reassignment.
cordoned.log.dirsDefault "" (none)
What it does
Log directories on a broker that should receive no new replicas. * cordons the whole broker.
When to change it
Set per broker before draining it for decommissioning. Remove it if the broker is to stay in service.
broker.rackDefault null
What it does
The rack or availability zone of a broker, used to spread a partition's replicas when they are placed.
When to change it
Set it on every broker, including new ones, before reassigning. A new broker without it can receive replicas that put two copies in one zone.
metadata.max.age.ms (clients)Default 300000 (5 minutes)
What it does
How often a client refreshes metadata even without an error, which is how it discovers new partitions and brokers.
When to change it
Sets the worst-case delay before producers use added partitions. Usually left alone; plan partition increases around it.
auto.offset.reset (consumer)Default latest
What it does
Where a group starts on a partition for which it has no committed offset.
When to change it
Decides whether records written to a newly added partition before the consumer finds it are read or skipped. For topics whose partition count may grow, do not leave this at latest without thinking about that gap.

Hands-on exercise

Troubleshoot: the restart script that refused orders

Work out from a script log and client symptoms why a routine rolling restart refused writes, reproduce the failure in the cluster explorer, then perform the restart correctly.

Steps · 0 of 7 done

Rolling restart of three brokers

Simulation · not a real cluster

orders.events with 3 partitions, replication factor 3, min.insync.replicas 2. First reproduce the failed restart, then run a correct one: stop, wait for the ISR, restart, next.

What can go wrong

Gating on a port, a process check or a sleep
Symptom
A rolling restart that has worked for months suddenly causes NotEnoughReplicasException for several minutes, usually on the day one broker takes longer than usual to come back.
Cause
The script moved to the next broker while the previous one was still recovering logs or catching up, so two replicas of many partitions were out of sync together.
Fix
Gate on kafka-topics.sh --describe --under-replicated-partitions returning nothing, with a timeout that stops the run and alerts a person.
Hard-killing brokers during a roll
Symptom
Each restart causes several seconds of producer timeouts instead of none, and brokers take much longer to return.
Cause
The stop timeout of the service manager is shorter than a controlled shutdown, so the process is killed. Leaders move only after the session timeout and the broker must recover its logs at startup.
Fix
Raise the stop timeout (systemd TimeoutStopSec, Kubernetes terminationGracePeriodSeconds) well beyond the longest observed shutdown, and confirm clean shutdown in the broker log.
Finalising in the same change as the software rollout
Symptom
A regression is found after an upgrade and the team discovers it cannot return to the previous version.
Cause
kafka-features.sh upgrade was run as soon as the last broker restarted. The cluster now uses a metadata version the old software may not read, and downgrade between those versions is not supported.
Fix
Make finalisation a separate, later change. Before it, read the upgrade notes for the exact versions involved.
Adding partitions to a keyed topic that is in use
Symptom
After a partition increase, a consumer applies events for the same order or account out of order, or per-key state stores look inconsistent.
Cause
The key-to-partition mapping changed with the partition count. New records for a key go to a different partition from its older, possibly unprocessed, records.
Fix
Size keyed topics for growth at creation. If more partitions are needed later, migrate to a new topic, or pause producers and let consumers drain the old partitions before the change.
Reassigning without a throttle, or leaving one behind
Symptom
Latency and lag rise cluster-wide during an expansion; or followers lag persistently some time after one.
Cause
Moving partitions copies their full contents and competes with client traffic. A throttle that is never cleared limits normal replication afterwards.
Fix
Always use --throttle, move in batches, and finish every reassignment with --verify.
Expecting a new broker to take load by itself
Symptom
Two brokers are added to an overloaded cluster and nothing improves; the new brokers sit idle.
Cause
Kafka assigns existing partitions to new brokers only when told to. Only topics created afterwards may place replicas there.
Fix
Follow the addition with a planned, throttled reassignment. Add capacity before the cluster is too busy to afford the copy.

How to troubleshoot

During a rolling restart, the gate never passes: under-replicated partitions stay above 0

Check

`kafka-topics.sh --bootstrap-server … --describe --under-replicated-partitions`; on the restarted broker `remainingLogsToRecover` and `kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica`
How to read it
If only the restarted broker is missing from the ISRs and its lag is falling, wait. Logs still being recovered mean the previous stop was not clean. If a different broker is missing, something else is wrong: stop the roll.
Producers get NotEnoughReplicasException during maintenance

Check

`kafka-topics.sh --bootstrap-server … --describe --under-min-isr-partitions`
How to read it
Two replicas of those partitions are out of sync at once: the broker being restarted plus one that had not finished catching up, or one that was already unhealthy. Stop the roll and let the missing replicas rejoin.
Stopping a broker takes very long, or it is killed by the service manager

Check

The broker's server log during shutdown for controlled shutdown progress; the service unit's stop timeout; topics with replication factor 1 via `kafka-topics.sh --describe`
How to read it
Controlled shutdown waits for leaderships to move, and cannot complete for partitions with no other live replica. Raise the timeout, and find single-replica topics before the maintenance.
After the roll, a few brokers handle most of the traffic

Check

`kafka.server:type=ReplicaManager,name=LeaderCount` per broker; `PreferredReplicaImbalanceCount` on the controller
How to read it
Leadership stayed where it moved during the restarts. It corrects itself within the controller's check interval if automatic rebalancing is on; otherwise run kafka-leader-election.sh --election-type preferred --all-topic-partitions.
Unsure whether an upgrade has been finalised, or whether the cluster can be rolled back

Check

`kafka-features.sh --bootstrap-server … describe`, comparing `FinalizedVersionLevel` with `SupportedMaxVersion` for `metadata.version`; the `CurrentMetadataVersion` metric
How to read it
Finalised level below the supported maximum: phase 1 only, rollback by reinstalling the old version is open. Equal: finalised, and the upgrade notes decide whether a downgrade exists.
A reassignment has been running for hours without finishing

Check

`kafka-reassign-partitions.sh --bootstrap-server … --reassignment-json-file plan.json --verify`; `ReassignmentBytesInPerSec` on the destination; `kafka.server:type=FetcherLagMetrics,name=ConsumerLag` for the moving replicas; `BytesInPerSec` for the topic
How to read it
Lag falling: it is progressing. Lag flat or rising while reassignment bytes sit at the throttle: the throttle is below the topic's write rate. Raise it with --execute --additional --throttle.
After adding partitions, a consumer group seems to have missed records

Check

`kafka-consumer-groups.sh --bootstrap-server … --describe --group …` for the new partitions' current offset and log end offset; the group's `auto.offset.reset`
How to read it
If the group's first committed offset on a new partition is above the partition's start, it began at the end and skipped what was produced before it discovered the partition. Those records are still in the log and can be reprocessed by resetting the group's offset for that partition.

Key takeaways

  • Every planned change is a rolling restart: one broker out, changed, back, fully in sync, then the next.
  • The gate between brokers is zero under-replicated partitions, asked of the cluster. A port, a process check or a timer is not a gate.
  • Always stop brokers with a controlled shutdown and give it time to finish.
  • An upgrade has two phases. New software can be rolled back; finalising metadata.version should be treated as one-way and done later, as its own change.
  • 4.x is KRaft only. KRaft clusters at 3.3 or later upgrade directly; ZooKeeper clusters must migrate to KRaft on a 3.x bridge release first.
  • New brokers are empty until you reassign partitions to them, with a throttle, and finish with --verify.
  • Adding partitions changes where keys go and cannot be undone. For keyed, ordered data, plan a migration instead.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

You are rolling a 6-broker cluster (replication factor 3, min.insync.replicas=2). Broker 3 was restarted 4 minutes ago. Its process is up, its port accepts connections, and it appears in the controller's active broker count. kafka-topics.sh --describe --under-replicated-partitions still prints 212 lines, all with broker 3 missing from Isr, and that number was 480 two minutes ago.

What do you do next?

Scenario 2 of 3

All brokers and controllers of a cluster were upgraded from 3.9 to 4.0 software this afternoon. kafka-features.sh describe shows metadata.version FinalizedVersionLevel 3.9-IV0 and SupportedMaxVersion 4.0-IV3. A colleague proposes finalising now 'to complete the change', before the overnight batch peak has run on the new version.

What is the better plan?

Scenario 3 of 3

Topic account.events has 12 partitions, keyed by account id, and its consumer applies events in order to build account balances. Lag is growing at peak because 12 consumers are not enough. Someone suggests kafka-topics.sh --alter --partitions 24 right now, with producers running and about 300,000 unprocessed records in the topic.

What is the main risk of doing that?

References

References: Apache Kafka documentation: Upgrading (rolling upgrade, finalisation, downgrade notes, 4.0 changes) ↗ · Apache Kafka documentation: Basic Kafka operations (graceful shutdown, modifying topics, expanding, decommissioning, throttling) ↗ · Apache Kafka documentation: Compatibility (client and broker matrix for 4.0) ↗ · Apache Kafka documentation: Operations, KRaft (feature versions, controller membership, ZooKeeper migration note) ↗ · Apache Kafka documentation: KRaft vs ZooKeeper (removed configurations and metrics) ↗ · Apache Kafka documentation: Broker configuration reference ↗ · Apache Kafka documentation: Java version ↗ · KIP-896: Remove old client protocol API versions in Kafka 4.0 ↗ · KIP-584: Versioning scheme for features ↗ · Report a problem