Path 8 · lesson 5 of 5
Cluster resources and scaling
How to size disk, network, memory and CPU for a Kafka cluster from throughput and retention, and what adding brokers, consumers or tiered storage really changes.
What you will learn
- Calculate the disk a cluster needs from its byte rate, retention, replication factor and a fill limit
- Estimate network traffic per broker, including replication and every consumer group
- Explain what the page cache does for Kafka and how to reason about broker memory
- Name what uses broker CPU and why TLS changes the cost of serving consumers
- Add brokers to a cluster and move partitions to them with a throttle, and estimate how long it takes
- Describe what tiered storage changes in the disk calculation and what its documented limits are
Before you start
The concept
A broker uses four resources, and each has a different driver. Sizing a cluster means working out each one from the workload, finding which runs out first, and leaving room for a broker to fail and for traffic to grow.
| Resource | What uses it | Rule of thumb for sizing |
|---|---|---|
| Disk | Every retained byte, once per replica | byte rate × retention time × replication factor, divided by the fill level you allow |
| Network | Produce traffic in, replication between brokers, one copy out per consumer group | produce rate × (1 + followers + consumer groups), shared over the brokers, at peak |
| Memory | The operating system's page cache holding recent log data; a moderate JVM heap | Enough cache to hold the data that producers just wrote and consumers are about to read |
| CPU | Request handling, validating compressed batches, TLS, and recompression if a topic forces a codec | Watch thread idle ratios; usually the last to run out without TLS |
Scaling has the same four dimensions plus one fact that surprises people: Kafka does not move existing data to a new broker by itself. A broker added to a cluster holds nothing until partitions are reassigned to it or new topics are created. Scaling a cluster is therefore two steps: add capacity, then move load onto it at a controlled rate.
Where replicas and leaders live
Simulation · not a real clusterFour brokers, one topic with six partitions and three replicas each. Count the replicas and leaders per broker, then stop a broker and count the leaders again.
Why it exists
Kafka's performance comes from using these resources in a particular way, and the sizing rules follow from that design. The log is written sequentially and never updated in place. Writes go into the operating system's page cache and are flushed in the background. Reads of recent data come straight from that cache, and the broker sends log data to the network without copying it through the application. The design documentation states the result: on a cluster where consumers are mostly caught up, there is no read activity on the disks at all.
That is also how a healthy cluster turns into a slow one without any change in produce rate. A consumer group that falls hours behind, or a job that replays old data, asks for segments that are no longer in memory. The broker now reads from disk, those reads compete with writes, and they push recent data out of the cache, which slows the live consumers too. Capacity planning has to cover this case and not only the steady state.
Replication is the other reason the numbers are larger than expected. With a replication factor of 3 every produced byte is stored three times and crosses the network between brokers twice. The cost buys the ability to lose a broker without losing data or availability, so it is not optional for business topics; it has to be in the arithmetic.
How it works inside
Disk
Use the byte rate as stored. Batches are kept on disk in the form the producer sent them, so a compressed batch costs its compressed size. Use the average rate over the retention period, not the peak, because disk integrates over time.
| Step | Arithmetic (assumed example) | Result |
|---|---|---|
| Average produce rate, all topics, as stored | Assumed | 20 MB/s |
| Per day | 20 MB/s × 86,400 s | 1,728 GB ≈ 1.73 TB |
| Retention of 7 days | 1.73 TB × 7 | 12.1 TB |
| Replication factor 3 | 12.1 TB × 3 | 36.3 TB |
| Allow disks to fill to 70% | 36.3 TB / 0.7 | 51.8 TB |
| Spread over 6 brokers | 51.8 TB / 6 | 8.6 TB per broker |
| Check: one broker lost and its replicas rebuilt on the other 5 | 36.3 TB / 5 = 7.3 TB of 8.6 TB | 84% full: workable, not comfortable |
Topics differ, so in practice you do this per topic and add up. A topic's own retention.ms overrides the broker default (log.retention.hours, 168). retention.bytes is a limit per partition: the topic reference says to multiply it by the partition count to get the topic's limit. Compacted topics are sized by the number of distinct keys and their value size, not by rate × time.
Network
| Flow | Arithmetic (same example) | Cluster total |
|---|---|---|
| Producers to leaders | 20 MB/s | 20 MB/s in |
| Leaders to followers, replication factor 3 | 20 MB/s × 2 followers | 40 MB/s out of leaders, 40 MB/s into followers |
| Leaders to consumers, 3 groups each reading everything | 20 MB/s × 3 | 60 MB/s out |
| Inbound, all brokers | 20 + 40 | 60 MB/s |
| Outbound, all brokers | 40 + 60 | 100 MB/s |
| Outbound per broker, 6 brokers, evenly spread | 100 / 6 = 16.7 MB/s × 8 | 133 Mbit/s |
| At a peak of 3× the average | 133 × 3 | 400 Mbit/s per broker |
The table assumes leaders are spread evenly. They are not always: after a broker restart its partitions are led by other brokers until leadership is moved back to the preferred replica, which the controller does periodically when auto.leader.rebalance.enable is true (the default). Broker metrics BytesInPerSec and BytesOutPerSec (per topic and in total) show the real distribution, with ReplicationBytesInPerSec and ReplicationBytesOutPerSec for the replication share.
Memory and the page cache
Kafka deliberately keeps log data out of the JVM heap. The design documentation gives the reasons: Java objects carry high memory overhead, garbage collection slows as the heap grows, and the operating system would cache the same files anyway. Leaving memory to the page cache gives a cache of nearly the whole machine that stays warm across a broker restart. So a broker wants a moderate heap and as much free memory as possible for the operating system.
The hardware guide gives a minimum: enough memory to buffer about 30 seconds of writes, computed as write throughput × 30. A more useful question is how far behind a consumer can be and still be served from memory:
| Quantity | Arithmetic (same example) | Result |
|---|---|---|
| Writes per broker, average | Inbound 60 MB/s / 6 brokers | 10 MB/s |
| Writes per broker at 3× peak | 10 × 3 | 30 MB/s |
| Documented minimum, 30 seconds of writes at peak | 30 MB/s × 30 s | 0.9 GB |
| Cache to cover consumers up to 10 minutes behind, at peak | 30 MB/s × 600 s | 18 GB |
| Cache to cover consumers up to 10 minutes behind, on average | 10 MB/s × 600 s | 6 GB |
CPU
- Request handling. Network threads (
num.network.threads, default 3 per listener) read requests and write responses; I/O threads (num.io.threads, default 8) process them. The monitoring guide says their idle ratios,NetworkProcessorAvgIdlePercentandRequestHandlerAvgIdlePercent, should ideally stay above 0.3. - Compression. The producer compresses and the consumer decompresses. The broker decompresses each batch to validate it and stores it as received. If a topic's
compression.typenames a codec different from the producer's, the broker recompresses every batch, which is avoidable CPU. The default,producer, keeps the producer's codec. - TLS. Encryption costs CPU on every byte in and out. It also removes the zero-copy path: the design documentation notes that
sendfileis not used when SSL is enabled, so data for consumers is copied through the broker process and encrypted there. Expect consumer-heavy workloads to cost noticeably more CPU with TLS, and measure it on your hardware. - Many small requests. A client that sends unbatched records, or thousands of clients with tiny fetches, costs CPU per request regardless of bytes.
Adding brokers
The operations guide is explicit: new servers are not automatically assigned any data partitions, so they do no work until partitions are moved to them or new topics are created. Moving is done with kafka-reassign-partitions.sh, in three modes: --generate proposes a plan for a list of topics and target brokers, --execute starts it, and --verify reports progress. The tool does not look at data sizes or load, so deciding what to move is the operator's job.
A reassignment works through normal replication. The new broker is added as a follower of the partition and copies its whole log. When it has caught up and joined the in-sync replicas, one of the old replicas deletes its copy. Copying competes with live traffic for disk and network, so you pass a throttle: an upper bound in bytes per second on replication traffic for the partitions being moved.
| Quantity | Arithmetic (same example) | Result |
|---|---|---|
| Data stored in the cluster | From the disk table | 36.3 TB |
| Brokers | 6, adding 2 | 8 |
| Even share per broker after the move | 36.3 TB / 8 | 4.54 TB |
| Data to copy to each new broker | Its even share | 4.54 TB = 4,540,000 MB |
| At a throttle of 50 MB/s | 4,540,000 / 50 = 90,800 s | about 25 hours |
| At a throttle of 100 MB/s | 4,540,000 / 100 = 45,400 s | about 12.6 hours |
| Network used on a new broker at 50 MB/s | 50 × 8 | 400 Mbit/s inbound |
Scaling consumers
Adding consumers to a group raises its throughput up to the partition count, at the cost of a rebalance each time membership changes. From the cluster's point of view, more consumers in one group do not add bytes: the same data is fetched over more connections. What adds load is another group reading the same topics, which is a full extra copy of the stream on the outbound side, and any consumer far behind the tail, which turns cache reads into disk reads.
Scale the group, then meet the next limit
Simulation · not a real clusterSix consumers on twelve partitions cannot keep up with 900 records per second. Add consumers, then raise the traffic and see which limit comes next.
Tiered storage, in outline
Tiered storage (KIP-405) splits a topic's log into two tiers. The local tier is the broker's disk, as before. The remote tier is an external system such as object storage, which receives completed log segments. Consumers reading the tail are served from the local tier and the page cache; a consumer that asks for old data is served from the remote tier.
- What it changes in sizing. Local disk is sized by
local.retention.msinstead ofretention.ms. In the example, one day of local retention is 1.73 TB × 3 = 5.2 TB on broker disks instead of 36.3 TB, with the full seven days held remotely. Reassignments and broker replacements also copy far less. - Status. The Kafka 4.4 documentation describes tiered storage as working for most use cases and links to its GA release notes. It is off by default:
remote.log.storage.system.enableon the broker andremote.storage.enableon the topic are both false. - What you must bring. Apache Kafka does not ship a
RemoteStorageManagerimplementation for any storage system. You configure one from a vendor or an open-source project throughremote.log.storage.manager.class.name. - Documented limitations. No support for compacted topics. Tiered storage must be disabled on all topics before it can be disabled on the broker. Reads of old data depend on the remote system's latency and cost.
Architecture
Drawing diagram…
Flowchart: producers send to partition leaders, which append to the page cache; the cache is flushed to local log segments and replicated to followers. Caught-up consumers read from the page cache, lagging consumers read from disk, completed segments may go to a remote tier, and a new empty broker receives data only through a throttled reassignment.
flowchart TB PROD["Producers 20 MB per second"] -->|"network in"| LEAD["Partition leaders"] LEAD -->|"append"| PC["Page cache on each broker"] PC -->|"background flush"| DISK["Local log segments, kept for retention"] LEAD -->|"replication, 2 followers"| FOL["Follower replicas on other brokers"] FOL --> PC PC -->|"tail reads, zero copy without TLS"| LIVE["Caught up consumer groups"] DISK -->|"disk reads"| LATE["Lagging or replaying consumers"] DISK -->|"completed segments, if tiered storage is on"| REMOTE["Remote tier"] NEW["New broker, empty"] -.->|"reassignment with throttle"| FOL
- Every produced byte enters once, is replicated to the followers, and leaves once per consumer group. That is the network calculation.
- The page cache sits between the network and the disk on both the write and the read path. Consumers at the tail never touch the disk.
- Lagging and replaying consumers take the lower path through the disk, which is what makes a backlog expensive for the whole broker.
- The dotted line is the only way a new broker gets existing data: a reassignment that an operator starts and throttles.
Message flow
Drawing diagram…
Sequence diagram: the operator generates and executes a reassignment plan with a throttle, the controller makes the new broker a follower, it fetches the log from the leader at the throttled rate, joins the in-sync replicas, the old replica deletes its copy, and a verify run confirms completion and removes the throttle.
sequenceDiagram
participant O as Operator
participant T as Reassign tool
participant C as Controller
participant N as New broker 7
participant L as Leader broker 2
participant X as Old replica broker 3
O->>T: generate plan for topics and brokers 7 and 8
T-->>O: current and proposed assignment
O->>T: execute with throttle 50 MB per second
T->>C: alter partition reassignments
C->>N: become follower of the partition
loop until caught up
N->>L: fetch at throttled rate
L-->>N: log data
end
L->>C: broker 7 joined the in-sync replicas
C->>X: remove replica
X->>X: delete local copy
O->>T: verify
T-->>O: completed, throttle removed- Generating a plan changes nothing. Save the current assignment it prints: it is the rollback plan.
- During the copy the partition has one replica more than its replication factor. Both the source and the destination hold the data, which is why free disk space is needed for a move.
- The partition stays available throughout. Producers and consumers keep using the leader.
- The final verify is part of the procedure, not an optional check: it is what removes the throttle settings from brokers and topics.
In a real system
A typical retail platform runs one cluster for its order, payment, inventory and clickstream topics. The clickstream dominates bytes; the order topics dominate importance. A capacity sheet lists each topic with its rate, retention and replication factor, and the cluster totals come from adding the rows. All numbers here are assumptions for the example.
| Topic | Avg rate as stored | Retention | RF | Disk = rate × 86,400 × days × RF |
|---|---|---|---|---|
orders.events | 0.5 MB/s | 7 days | 3 | 0.5 × 86,400 × 7 × 3 = 907 GB |
payments.events | 0.3 MB/s | 14 days | 3 | 0.3 × 86,400 × 14 × 3 = 1,089 GB |
inventory.events | 0.7 MB/s | 7 days | 3 | 0.7 × 86,400 × 7 × 3 = 1,270 GB |
web.clicks (zstd) | 6 MB/s | 3 days | 3 | 6 × 86,400 × 3 × 3 = 4,666 GB |
| Total | 7.5 MB/s | 7,932 GB ≈ 7.9 TB | ||
| At a 70% fill limit | 7.9 / 0.7 = 11.3 TB | |||
| Per broker, 4 brokers | 2.8 TB |
Two decisions come out of a sheet like this. Retention is a cost setting per topic and deserves a reason: the order topics keep 7 days so that a consumer outage over a long weekend can be recovered, while the clickstream keeps 3 because its consumers can be rebuilt from the data warehouse. And the first resource to run out is visible in advance: here it is disk, driven by one topic, so the alert that matters is disk fill per broker and the lever is that topic's retention or tiered storage.
Java and Spring Boot
The commands for the three operations in this lesson: seeing what uses the disk, expanding the cluster with a throttled reassignment, and enabling tiered storage on a topic once the brokers are configured for it.
# Size of every replica of a topic on brokers 1 to 4 (JSON output, sizes in bytes).bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe \ --broker-list 1,2,3,4 --topic-list web.clicks# Retention and other overrides set on the topic.bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe \ --entity-type topics --entity-name web.clicks# Change retention for one topic to 3 days (259,200,000 ms).bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter \ --entity-type topics --entity-name web.clicks \ --add-config retention.ms=259200000Why it is written this way
kafka-log-dirs.shreports, per broker and log directory, each partition replica with its size. Summing by topic shows which topics own the disk.- Lowering
retention.msfrees space only after the next retention check deletes whole segments that are now too old; it is not immediate. - The same per-partition size is available continuously as the broker metric
kafka.log:type=Log,name=Size,topic=…,partition=….
# 1. Which topics to spread onto the enlarged broker set.cat > topics-to-move.json <<'JSON'{ "topics": [ { "topic": "web.clicks" }, { "topic": "orders.events" } ], "version": 1 }JSON# 2. Propose an assignment over old and new brokers. Nothing moves yet.bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \ --topics-to-move-json-file topics-to-move.json \ --broker-list "1,2,3,4,5,6,7,8" --generate# Save the "Current partition replica assignment" as rollback.json# and the "Proposed partition reassignment configuration" as expand.json.# 3. Start the move, limited to 50 MB/s of replication traffic.bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \ --reassignment-json-file expand.json --execute --throttle 50000000# 4. Raise the throttle on a running move if it is safe to go faster.bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \ --reassignment-json-file expand.json --additional --execute --throttle 100000000# 5. Check progress. Once everything is complete, this run also removes the throttle.bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \ --reassignment-json-file expand.json --verifyWhy it is written this way
--broker-listin the generate step is the full target set. The tool spreads the listed topics evenly over those brokers by partition count; it does not consider partition sizes or current load.- Move in batches of a few topics, largest last, so that each step is short and can be verified. A plan that moves everything at once runs for the whole duration with no checkpoint.
--throttleis in bytes per second. It must exceed the write rate of the partitions being moved, or the new replicas never catch up.- Watch
kafka.server:type=ReplicaManager,name=ReassigningPartitionsand theReassignmentBytesInPerSec/ReassignmentBytesOutPerSecbroker metrics during the move, and under-replicated partitions for collateral effects. - Always finish with
--verifyuntil it reports completion: that run clears the throttle configuration.
# Requires brokers started with remote.log.storage.system.enable=true and a# RemoteStorageManager plugin configured (Apache Kafka does not ship one).# Keep 1 day on broker disks and 7 days in total for the clickstream.bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter \ --entity-type topics --entity-name web.clicks \ --add-config 'remote.storage.enable=true,local.retention.ms=86400000,retention.ms=604800000'Why it is written this way
retention.msis the total retention across both tiers;local.retention.msis how long a segment stays on broker disk. The default of -2 means the local value followsretention.ms.- A local segment becomes eligible for deletion only after it has been uploaded, so a remote storage outage makes local disk grow. Monitor the copy lag (
RemoteCopyLagBytes). - Not available for compacted topics. Test reads of old data through the remote tier before relying on it for recovery.
Configuration
Broker and topic settings behind the sizing rules, with defaults from the Kafka 4.4 references.
log.retention.hours (broker) / retention.ms (topic)Default 168 / 604800000 (7 days)
- What it does
- How long a log segment is kept before it may be deleted under the delete policy. The topic setting overrides the broker default.
- When to change it
- Set per topic from how long consumers may need to re-read. It multiplies directly into disk.
log.retention.bytes (broker) / retention.bytes (topic)Default -1 (no size limit)
- What it does
- Maximum size of a partition's log before old segments are deleted. The limit is per partition.
- When to change it
- Use it as a safety cap on topics whose rate can spike. Remember that it shortens how far back consumers can read.
log.segment.bytes (broker) / segment.bytes (topic)Default 1073741824 (1 GiB)
- What it does
- Size at which the active segment is closed and a new one started. Retention deletes whole closed segments.
- When to change it
- Retention and cleaning work one file at a time, so larger segments mean fewer files and coarser retention. The 4.4 topic reference notes that a new segment is also rolled once
retention.msis satisfied; check the behaviour of your version on low-volume topics.
default.replication.factor (broker)Default 1
- What it does
- Replication factor for topics created without one.
- When to change it
- Set it to 3 on a production cluster, or always create topics explicitly. The default gives no redundancy.
num.network.threads / num.io.threads (broker)Default 3 / 8
- What it does
- Threads that read and write the network (per listener) and threads that process requests.
- When to change it
- Raise them when the idle ratios stay low and the host has spare cores. They do not help when disk or network is the limit.
num.replica.fetchers (broker)Default 1
- What it does
- Fetcher threads used to replicate from each source broker.
- When to change it
- Raise it if followers lag at high throughput while disks and network have capacity. It costs CPU and memory on both sides.
leader.replication.throttled.rate / follower.replication.throttled.rate (broker, dynamic)Default 9223372036854775807 (unlimited)
- What it does
- Upper bound in bytes per second on replication traffic for the replicas listed as throttled on each topic. Set by the reassignment tool's
--throttle. - When to change it
- Set through the tool, not by hand. Keep it above the write rate, and make sure it is cleared with
--verifyafterwards.
auto.leader.rebalance.enable (broker)Default true
- What it does
- A background check moves leadership back to the preferred replica when leaders are unevenly distributed.
- When to change it
- Leave it on. Without it, a restarted broker stays without leaders and the others carry its share of produce and fetch traffic.
remote.log.storage.system.enable (broker)Default false
- What it does
- Enables the tiered storage services on a broker. A
RemoteStorageManagerimplementation must be configured as well. - When to change it
- Enable it when long retention is driving disk cost and a storage plugin you trust is available.
remote.storage.enable / local.retention.ms (topic)Default false / -2
- What it does
- Turns tiered storage on for a topic, and sets how long segments stay on local disk. -2 means the local value follows
retention.ms. - When to change it
- Choose a local retention that covers normal consumer lag and recovery, so that only real replays go to the remote tier.
Hands-on exercise
Size a cluster on paper, then scale its consumers
Do the disk, network and reassignment arithmetic for a small cluster by hand, then use the throughput model to see what scaling a consumer group can and cannot do.
Steps · 0 of 6 done
Scaling a consumer group step by step
Simulation · not a real clusterA simple queueing model with illustrative numbers. Each consumer handles 100 records per second and they share a database pool of 20 connections.
What you should see
Disk: 8 × 86,400 × 3 × 3 = 6,221 GB; / 0.65 = 9,570 GB; / 4 = about 2.4 TB per broker. Network: 8 × 2 followers + 8 × 2 groups = 32 MB/s out, 8 MB/s per broker, 64 Mbit/s. Expansion: 6,221 / 5 = 1,244 GB; at 30 MB/s that is 41,470 s, about 11.5 hours. Model: 6 consumers give 600 for 900 arriving (too few consumers); 12 give 1,200 with 25% headroom. At 1,500 processing time is the limit because every partition has a consumer. With 24 and 24 the ceiling is 2,000, set by the 20 database connections.
What can go wrong
Adding brokers and expecting relief
- Symptom
- Two brokers were added to a cluster whose disks were at 85%. A day later the old brokers are at 86% and the new ones are empty.
- Cause
- New brokers receive no existing partitions automatically. Only topics created afterwards may be placed on them.
- Fix
- Follow the addition with a throttled partition reassignment, moving the largest topics in batches, and verify each batch.
Sizing disk without replication or headroom
- Symptom
- The estimate said 12 TB. The cluster ran out of disk in the first month.
- Cause
- The calculation was rate × retention only. It left out the replication factor of 3, the fill limit, and the segments that retention has not yet deleted.
- Fix
- Disk = rate × retention × replication factor / fill limit, per topic, then summed. Recalculate when a topic's rate or retention changes.
A throttle below the write rate
- Symptom
- A reassignment has shown the same partitions in progress for two days. The new replicas' lag is not shrinking.
- Cause
- The partitions receive more bytes per second than the throttle allows the new replica to copy, so it never catches up.
- Fix
- Compare the throttle with
BytesInPerSecfor the topics being moved and raise it with--additional --execute --throttle. Watch the replica lag fall.
A throttle left in place
- Symptom
- Weeks after an expansion, followers of some topics lag during peaks and partitions flap in and out of the in-sync set.
- Cause
- The reassignment finished and
--verifywas never run, so the throttle still limits normal replication for those replicas. - Fix
- Run
--verifywith the same JSON file until it reports that throttles were cleared. Checkkafka-configs.sh --describe --entity-type brokersfor leftover throttle settings.
A replay that slows the live consumers
- Symptom
- A team re-reads a month of history. During the replay, produce latency and live consumers' fetch latency rise across unrelated topics.
- Cause
- Old segments are read from disk and fill the page cache, evicting the recent data that live consumers are normally served from. Disk reads now compete with writes.
- Fix
- Give replay clients a
consumer_byte_ratequota, run them off-peak, and size memory for the lag you intend to support.
How to troubleshoot
Disk usage on brokers is climbing and you need to know which topics own it
Check
bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe --broker-list 1,2,3,4
- How to read it
- Lists each partition replica with its size in bytes per log directory. Sum by topic. The largest topics, multiplied by their retention, are where a change has an effect.
You need to know whether a reassignment is still running
Check
bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file expand.json --verify; metric kafka.server:type=ReplicaManager,name=ReassigningPartitions
- How to read it
- The tool lists each partition as completed or still in progress. The metric is non-zero on brokers leading partitions that are being moved.
Brokers respond slowly under load
Check
kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent and kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent
- How to read it
- Both are between 0 and 1 and should ideally stay above 0.3. Low request handler idle points at CPU or slow disk I/O in request processing; low network processor idle points at network threads, often with TLS.
One broker carries much more traffic than the others
Check
kafka.server:type=ReplicaManager,name=LeaderCount and name=PartitionCount per broker; kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
- How to read it
- A high leader count relative to partition count means leadership is uneven, typically after restarts. A high partition count means replicas are uneven and need reassignment.
After adding capacity, replication seems unhealthy
Check
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions; bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions
- How to read it
- Should be 0 in steady state. During a reassignment some partitions show extra replicas catching up. Persistent under-replication afterwards suggests a leftover throttle or an overloaded broker.
Tiered storage is on and local disks keep growing
Check
kafka.server:type=BrokerTopicMetrics,name=RemoteCopyLagBytes,topic=…
- How to read it
- Bytes in closed segments not yet copied to the remote tier. A growing value means uploads are failing or too slow, and local segments cannot be deleted until they are uploaded.
Key takeaways
- Disk = byte rate as stored × retention × replication factor, divided by the fill level you allow. Do it per topic and add up.
- Every produced byte crosses the network once in, once per follower, and once per consumer group. Size for the peak.
- Kafka relies on the page cache: caught-up consumers are served from memory, lagging and replaying consumers from disk.
- TLS and forced recompression are the avoidable or unavoidable CPU costs; watch the request handler and network processor idle ratios.
- A new broker holds nothing until partitions are reassigned to it. Reassign with a throttle above the write rate, and clear it with
--verify. - Scaling a consumer group is capped by partitions and by its dependencies; a new group is a full extra copy of outbound traffic.
- Tiered storage moves old segments off broker disks. It is off by default, needs a storage plugin Kafka does not ship, and does not support compacted topics.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A new cluster will store an average of 12 MB/s as written by producers after compression. Retention is 5 days on every topic, the replication factor is 3, and the team does not want any disk above 60% full. They plan to run 6 brokers.
About how much disk does each broker need?
Scenario 2 of 3
A 5-broker cluster has disks at 82%. The team adds 3 brokers on Monday. On Wednesday the original five are at 83% and the new three show almost no disk use and no network traffic. No topics were created in between.
What explains this, and what is the next step?
Scenario 3 of 3
During a cluster expansion a reassignment was started with --throttle 20000000 (20 MB/s). After 18 hours, --verify still reports the partitions of one large topic as in progress and the new replicas' lag is flat. That topic's BytesInPerSec peaks at 35 MB/s during the day. Other, smaller topics in the same plan completed within hours.
What should the operator do?
References
References: Apache Kafka 4.4: Hardware and OS (memory estimate, disks, file descriptors, flush behaviour) ↗ · Apache Kafka 4.4: Basic Kafka operations (expanding your cluster, partition reassignment, limiting bandwidth during data migration) ↗ · Apache Kafka 4.4: Design (persistence, page cache, efficiency, zero-copy and TLS, batch compression) ↗ · Apache Kafka 4.4: Monitoring (broker byte rates, idle ratios, reassignment and tiered storage metrics) ↗ · Apache Kafka 4.4: Tiered storage (overview, configuration, limitations) ↗ · Apache Kafka 4.4: Broker configs (retention, segments, threads, replication throttles) ↗ · Apache Kafka 4.4: Topic configs (retention.ms, retention.bytes, segment.bytes, remote.storage.enable, local.retention.ms) ↗ · KIP-405: Kafka Tiered Storage ↗ · Report a problem