Skip to content

Path 8 · lesson 5 of 5

Advanced28 minApache Kafka 4.4 (operations: hardware and OS, basic operations, monitoring, tiered storage; design: efficiency; broker and topic configuration references). Kafka 4.x runs in KRaft mode only.

Cluster resources and scaling

How to size disk, network, memory and CPU for a Kafka cluster from throughput and retention, and what adding brokers, consumers or tiered storage really changes.

What you will learn

  • Calculate the disk a cluster needs from its byte rate, retention, replication factor and a fill limit
  • Estimate network traffic per broker, including replication and every consumer group
  • Explain what the page cache does for Kafka and how to reason about broker memory
  • Name what uses broker CPU and why TLS changes the cost of serving consumers
  • Add brokers to a cluster and move partitions to them with a throttle, and estimate how long it takes
  • Describe what tiered storage changes in the disk calculation and what its documented limits are

Before you start

The concept

A broker uses four resources, and each has a different driver. Sizing a cluster means working out each one from the workload, finding which runs out first, and leaving room for a broker to fail and for traffic to grow.

ResourceWhat uses itRule of thumb for sizing
DiskEvery retained byte, once per replicabyte rate × retention time × replication factor, divided by the fill level you allow
NetworkProduce traffic in, replication between brokers, one copy out per consumer groupproduce rate × (1 + followers + consumer groups), shared over the brokers, at peak
MemoryThe operating system's page cache holding recent log data; a moderate JVM heapEnough cache to hold the data that producers just wrote and consumers are about to read
CPURequest handling, validating compressed batches, TLS, and recompression if a topic forces a codecWatch thread idle ratios; usually the last to run out without TLS

Scaling has the same four dimensions plus one fact that surprises people: Kafka does not move existing data to a new broker by itself. A broker added to a cluster holds nothing until partitions are reassigned to it or new topics are created. Scaling a cluster is therefore two steps: add capacity, then move load onto it at a controlled rate.

Where replicas and leaders live

Simulation · not a real cluster

Four brokers, one topic with six partitions and three replicas each. Count the replicas and leaders per broker, then stop a broker and count the leaders again.

Why it exists

Kafka's performance comes from using these resources in a particular way, and the sizing rules follow from that design. The log is written sequentially and never updated in place. Writes go into the operating system's page cache and are flushed in the background. Reads of recent data come straight from that cache, and the broker sends log data to the network without copying it through the application. The design documentation states the result: on a cluster where consumers are mostly caught up, there is no read activity on the disks at all.

That is also how a healthy cluster turns into a slow one without any change in produce rate. A consumer group that falls hours behind, or a job that replays old data, asks for segments that are no longer in memory. The broker now reads from disk, those reads compete with writes, and they push recent data out of the cache, which slows the live consumers too. Capacity planning has to cover this case and not only the steady state.

Replication is the other reason the numbers are larger than expected. With a replication factor of 3 every produced byte is stored three times and crosses the network between brokers twice. The cost buys the ability to lose a broker without losing data or availability, so it is not optional for business topics; it has to be in the arithmetic.

How it works inside

Disk

Use the byte rate as stored. Batches are kept on disk in the form the producer sent them, so a compressed batch costs its compressed size. Use the average rate over the retention period, not the peak, because disk integrates over time.

StepArithmetic (assumed example)Result
Average produce rate, all topics, as storedAssumed20 MB/s
Per day20 MB/s × 86,400 s1,728 GB ≈ 1.73 TB
Retention of 7 days1.73 TB × 712.1 TB
Replication factor 312.1 TB × 336.3 TB
Allow disks to fill to 70%36.3 TB / 0.751.8 TB
Spread over 6 brokers51.8 TB / 68.6 TB per broker
Check: one broker lost and its replicas rebuilt on the other 536.3 TB / 5 = 7.3 TB of 8.6 TB84% full: workable, not comfortable
The 30% left free covers growth, segments that the periodic retention check has not yet deleted, and the double copy that exists while a partition is being moved.

Topics differ, so in practice you do this per topic and add up. A topic's own retention.ms overrides the broker default (log.retention.hours, 168). retention.bytes is a limit per partition: the topic reference says to multiply it by the partition count to get the topic's limit. Compacted topics are sized by the number of distinct keys and their value size, not by rate × time.

Network

FlowArithmetic (same example)Cluster total
Producers to leaders20 MB/s20 MB/s in
Leaders to followers, replication factor 320 MB/s × 2 followers40 MB/s out of leaders, 40 MB/s into followers
Leaders to consumers, 3 groups each reading everything20 MB/s × 360 MB/s out
Inbound, all brokers20 + 4060 MB/s
Outbound, all brokers40 + 60100 MB/s
Outbound per broker, 6 brokers, evenly spread100 / 6 = 16.7 MB/s × 8133 Mbit/s
At a peak of 3× the average133 × 3400 Mbit/s per broker
In this example a 1 Gbit/s interface is 40% used at peak before any catch-up or reassignment traffic. Every additional consumer group that reads the full stream adds another 20 MB/s of outbound traffic.

The table assumes leaders are spread evenly. They are not always: after a broker restart its partitions are led by other brokers until leadership is moved back to the preferred replica, which the controller does periodically when auto.leader.rebalance.enable is true (the default). Broker metrics BytesInPerSec and BytesOutPerSec (per topic and in total) show the real distribution, with ReplicationBytesInPerSec and ReplicationBytesOutPerSec for the replication share.

Memory and the page cache

Kafka deliberately keeps log data out of the JVM heap. The design documentation gives the reasons: Java objects carry high memory overhead, garbage collection slows as the heap grows, and the operating system would cache the same files anyway. Leaving memory to the page cache gives a cache of nearly the whole machine that stays warm across a broker restart. So a broker wants a moderate heap and as much free memory as possible for the operating system.

The hardware guide gives a minimum: enough memory to buffer about 30 seconds of writes, computed as write throughput × 30. A more useful question is how far behind a consumer can be and still be served from memory:

QuantityArithmetic (same example)Result
Writes per broker, averageInbound 60 MB/s / 6 brokers10 MB/s
Writes per broker at 3× peak10 × 330 MB/s
Documented minimum, 30 seconds of writes at peak30 MB/s × 30 s0.9 GB
Cache to cover consumers up to 10 minutes behind, at peak30 MB/s × 600 s18 GB
Cache to cover consumers up to 10 minutes behind, on average10 MB/s × 600 s6 GB
An estimate that treats the cache as holding the most recent writes. Consumers further behind than the cache reaches are served from disk.

CPU

  • Request handling. Network threads (num.network.threads, default 3 per listener) read requests and write responses; I/O threads (num.io.threads, default 8) process them. The monitoring guide says their idle ratios, NetworkProcessorAvgIdlePercent and RequestHandlerAvgIdlePercent, should ideally stay above 0.3.
  • Compression. The producer compresses and the consumer decompresses. The broker decompresses each batch to validate it and stores it as received. If a topic's compression.type names a codec different from the producer's, the broker recompresses every batch, which is avoidable CPU. The default, producer, keeps the producer's codec.
  • TLS. Encryption costs CPU on every byte in and out. It also removes the zero-copy path: the design documentation notes that sendfile is not used when SSL is enabled, so data for consumers is copied through the broker process and encrypted there. Expect consumer-heavy workloads to cost noticeably more CPU with TLS, and measure it on your hardware.
  • Many small requests. A client that sends unbatched records, or thousands of clients with tiny fetches, costs CPU per request regardless of bytes.

Adding brokers

The operations guide is explicit: new servers are not automatically assigned any data partitions, so they do no work until partitions are moved to them or new topics are created. Moving is done with kafka-reassign-partitions.sh, in three modes: --generate proposes a plan for a list of topics and target brokers, --execute starts it, and --verify reports progress. The tool does not look at data sizes or load, so deciding what to move is the operator's job.

A reassignment works through normal replication. The new broker is added as a follower of the partition and copies its whole log. When it has caught up and joined the in-sync replicas, one of the old replicas deletes its copy. Copying competes with live traffic for disk and network, so you pass a throttle: an upper bound in bytes per second on replication traffic for the partitions being moved.

QuantityArithmetic (same example)Result
Data stored in the clusterFrom the disk table36.3 TB
Brokers6, adding 28
Even share per broker after the move36.3 TB / 84.54 TB
Data to copy to each new brokerIts even share4.54 TB = 4,540,000 MB
At a throttle of 50 MB/s4,540,000 / 50 = 90,800 sabout 25 hours
At a throttle of 100 MB/s4,540,000 / 100 = 45,400 sabout 12.6 hours
Network used on a new broker at 50 MB/s50 × 8400 Mbit/s inbound
Expanding this cluster is a job of a day or more. Plan it before disks reach their fill limit, not after.

Scaling consumers

Adding consumers to a group raises its throughput up to the partition count, at the cost of a rebalance each time membership changes. From the cluster's point of view, more consumers in one group do not add bytes: the same data is fetched over more connections. What adds load is another group reading the same topics, which is a full extra copy of the stream on the outbound side, and any consumer far behind the tail, which turns cache reads into disk reads.

Scale the group, then meet the next limit

Simulation · not a real cluster

Six consumers on twelve partitions cannot keep up with 900 records per second. Add consumers, then raise the traffic and see which limit comes next.

Tiered storage, in outline

Tiered storage (KIP-405) splits a topic's log into two tiers. The local tier is the broker's disk, as before. The remote tier is an external system such as object storage, which receives completed log segments. Consumers reading the tail are served from the local tier and the page cache; a consumer that asks for old data is served from the remote tier.

  • What it changes in sizing. Local disk is sized by local.retention.ms instead of retention.ms. In the example, one day of local retention is 1.73 TB × 3 = 5.2 TB on broker disks instead of 36.3 TB, with the full seven days held remotely. Reassignments and broker replacements also copy far less.
  • Status. The Kafka 4.4 documentation describes tiered storage as working for most use cases and links to its GA release notes. It is off by default: remote.log.storage.system.enable on the broker and remote.storage.enable on the topic are both false.
  • What you must bring. Apache Kafka does not ship a RemoteStorageManager implementation for any storage system. You configure one from a vendor or an open-source project through remote.log.storage.manager.class.name.
  • Documented limitations. No support for compacted topics. Tiered storage must be disabled on all topics before it can be disabled on the broker. Reads of old data depend on the remote system's latency and cost.

Architecture

Drawing diagram…

Flowchart: producers send to partition leaders, which append to the page cache; the cache is flushed to local log segments and replicated to followers. Caught-up consumers read from the page cache, lagging consumers read from disk, completed segments may go to a remote tier, and a new empty broker receives data only through a throttled reassignment.

flowchart TB
  PROD["Producers 20 MB per second"] -->|"network in"| LEAD["Partition leaders"]
  LEAD -->|"append"| PC["Page cache on each broker"]
  PC -->|"background flush"| DISK["Local log segments, kept for retention"]
  LEAD -->|"replication, 2 followers"| FOL["Follower replicas on other brokers"]
  FOL --> PC
  PC -->|"tail reads, zero copy without TLS"| LIVE["Caught up consumer groups"]
  DISK -->|"disk reads"| LATE["Lagging or replaying consumers"]
  DISK -->|"completed segments, if tiered storage is on"| REMOTE["Remote tier"]
  NEW["New broker, empty"] -.->|"reassignment with throttle"| FOL
The four resources on one picture: network in and out, page cache, disk, and the paths that cost CPU and disk reads.
  • Every produced byte enters once, is replicated to the followers, and leaves once per consumer group. That is the network calculation.
  • The page cache sits between the network and the disk on both the write and the read path. Consumers at the tail never touch the disk.
  • Lagging and replaying consumers take the lower path through the disk, which is what makes a backlog expensive for the whole broker.
  • The dotted line is the only way a new broker gets existing data: a reassignment that an operator starts and throttles.

Message flow

Drawing diagram…

Sequence diagram: the operator generates and executes a reassignment plan with a throttle, the controller makes the new broker a follower, it fetches the log from the leader at the throttled rate, joins the in-sync replicas, the old replica deletes its copy, and a verify run confirms completion and removes the throttle.

sequenceDiagram
  participant O as Operator
  participant T as Reassign tool
  participant C as Controller
  participant N as New broker 7
  participant L as Leader broker 2
  participant X as Old replica broker 3
  O->>T: generate plan for topics and brokers 7 and 8
  T-->>O: current and proposed assignment
  O->>T: execute with throttle 50 MB per second
  T->>C: alter partition reassignments
  C->>N: become follower of the partition
  loop until caught up
    N->>L: fetch at throttled rate
    L-->>N: log data
  end
  L->>C: broker 7 joined the in-sync replicas
  C->>X: remove replica
  X->>X: delete local copy
  O->>T: verify
  T-->>O: completed, throttle removed
Moving one partition to a new broker: copy under a throttle, join the in-sync replicas, drop the old copy, clear the throttle.
  1. Generating a plan changes nothing. Save the current assignment it prints: it is the rollback plan.
  2. During the copy the partition has one replica more than its replication factor. Both the source and the destination hold the data, which is why free disk space is needed for a move.
  3. The partition stays available throughout. Producers and consumers keep using the leader.
  4. The final verify is part of the procedure, not an optional check: it is what removes the throttle settings from brokers and topics.

In a real system

A typical retail platform runs one cluster for its order, payment, inventory and clickstream topics. The clickstream dominates bytes; the order topics dominate importance. A capacity sheet lists each topic with its rate, retention and replication factor, and the cluster totals come from adding the rows. All numbers here are assumptions for the example.

TopicAvg rate as storedRetentionRFDisk = rate × 86,400 × days × RF
orders.events0.5 MB/s7 days30.5 × 86,400 × 7 × 3 = 907 GB
payments.events0.3 MB/s14 days30.3 × 86,400 × 14 × 3 = 1,089 GB
inventory.events0.7 MB/s7 days30.7 × 86,400 × 7 × 3 = 1,270 GB
web.clicks (zstd)6 MB/s3 days36 × 86,400 × 3 × 3 = 4,666 GB
Total7.5 MB/s7,932 GB ≈ 7.9 TB
At a 70% fill limit7.9 / 0.7 = 11.3 TB
Per broker, 4 brokers2.8 TB
The clickstream is 80% of the bytes with the shortest retention. Raising its retention from 3 to 7 days would add 6,221 GB, more than everything else together.

Two decisions come out of a sheet like this. Retention is a cost setting per topic and deserves a reason: the order topics keep 7 days so that a consumer outage over a long weekend can be recovered, while the clickstream keeps 3 because its consumers can be rebuilt from the data warehouse. And the first resource to run out is visible in advance: here it is disk, driven by one topic, so the alert that matters is disk fill per broker and the lever is that topic's retention or tiered storage.

Java and Spring Boot

The commands for the three operations in this lesson: seeing what uses the disk, expanding the cluster with a throttled reassignment, and enabling tiered storage on a topic once the brokers are configured for it.

disk-usage.shShell
# Size of every replica of a topic on brokers 1 to 4 (JSON output, sizes in bytes).bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe \  --broker-list 1,2,3,4 --topic-list web.clicks# Retention and other overrides set on the topic.bin/kafka-configs.sh --bootstrap-server localhost:9092 --describe \  --entity-type topics --entity-name web.clicks# Change retention for one topic to 3 days (259,200,000 ms).bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter \  --entity-type topics --entity-name web.clicks \  --add-config retention.ms=259200000

Why it is written this way

  • kafka-log-dirs.sh reports, per broker and log directory, each partition replica with its size. Summing by topic shows which topics own the disk.
  • Lowering retention.ms frees space only after the next retention check deletes whole segments that are now too old; it is not immediate.
  • The same per-partition size is available continuously as the broker metric kafka.log:type=Log,name=Size,topic=…,partition=….

Configuration

Broker and topic settings behind the sizing rules, with defaults from the Kafka 4.4 references.

log.retention.hours (broker) / retention.ms (topic)Default 168 / 604800000 (7 days)
What it does
How long a log segment is kept before it may be deleted under the delete policy. The topic setting overrides the broker default.
When to change it
Set per topic from how long consumers may need to re-read. It multiplies directly into disk.
log.retention.bytes (broker) / retention.bytes (topic)Default -1 (no size limit)
What it does
Maximum size of a partition's log before old segments are deleted. The limit is per partition.
When to change it
Use it as a safety cap on topics whose rate can spike. Remember that it shortens how far back consumers can read.
log.segment.bytes (broker) / segment.bytes (topic)Default 1073741824 (1 GiB)
What it does
Size at which the active segment is closed and a new one started. Retention deletes whole closed segments.
When to change it
Retention and cleaning work one file at a time, so larger segments mean fewer files and coarser retention. The 4.4 topic reference notes that a new segment is also rolled once retention.ms is satisfied; check the behaviour of your version on low-volume topics.
default.replication.factor (broker)Default 1
What it does
Replication factor for topics created without one.
When to change it
Set it to 3 on a production cluster, or always create topics explicitly. The default gives no redundancy.
num.network.threads / num.io.threads (broker)Default 3 / 8
What it does
Threads that read and write the network (per listener) and threads that process requests.
When to change it
Raise them when the idle ratios stay low and the host has spare cores. They do not help when disk or network is the limit.
num.replica.fetchers (broker)Default 1
What it does
Fetcher threads used to replicate from each source broker.
When to change it
Raise it if followers lag at high throughput while disks and network have capacity. It costs CPU and memory on both sides.
leader.replication.throttled.rate / follower.replication.throttled.rate (broker, dynamic)Default 9223372036854775807 (unlimited)
What it does
Upper bound in bytes per second on replication traffic for the replicas listed as throttled on each topic. Set by the reassignment tool's --throttle.
When to change it
Set through the tool, not by hand. Keep it above the write rate, and make sure it is cleared with --verify afterwards.
auto.leader.rebalance.enable (broker)Default true
What it does
A background check moves leadership back to the preferred replica when leaders are unevenly distributed.
When to change it
Leave it on. Without it, a restarted broker stays without leaders and the others carry its share of produce and fetch traffic.
remote.log.storage.system.enable (broker)Default false
What it does
Enables the tiered storage services on a broker. A RemoteStorageManager implementation must be configured as well.
When to change it
Enable it when long retention is driving disk cost and a storage plugin you trust is available.
remote.storage.enable / local.retention.ms (topic)Default false / -2
What it does
Turns tiered storage on for a topic, and sets how long segments stay on local disk. -2 means the local value follows retention.ms.
When to change it
Choose a local retention that covers normal consumer lag and recovery, so that only real replays go to the remote tier.

Hands-on exercise

Size a cluster on paper, then scale its consumers

Do the disk, network and reassignment arithmetic for a small cluster by hand, then use the throughput model to see what scaling a consumer group can and cannot do.

Steps · 0 of 6 done

Scaling a consumer group step by step

Simulation · not a real cluster

A simple queueing model with illustrative numbers. Each consumer handles 100 records per second and they share a database pool of 20 connections.

Take it further in the lab: Consumer lag during high load

What can go wrong

Adding brokers and expecting relief
Symptom
Two brokers were added to a cluster whose disks were at 85%. A day later the old brokers are at 86% and the new ones are empty.
Cause
New brokers receive no existing partitions automatically. Only topics created afterwards may be placed on them.
Fix
Follow the addition with a throttled partition reassignment, moving the largest topics in batches, and verify each batch.
Sizing disk without replication or headroom
Symptom
The estimate said 12 TB. The cluster ran out of disk in the first month.
Cause
The calculation was rate × retention only. It left out the replication factor of 3, the fill limit, and the segments that retention has not yet deleted.
Fix
Disk = rate × retention × replication factor / fill limit, per topic, then summed. Recalculate when a topic's rate or retention changes.
A throttle below the write rate
Symptom
A reassignment has shown the same partitions in progress for two days. The new replicas' lag is not shrinking.
Cause
The partitions receive more bytes per second than the throttle allows the new replica to copy, so it never catches up.
Fix
Compare the throttle with BytesInPerSec for the topics being moved and raise it with --additional --execute --throttle. Watch the replica lag fall.
A throttle left in place
Symptom
Weeks after an expansion, followers of some topics lag during peaks and partitions flap in and out of the in-sync set.
Cause
The reassignment finished and --verify was never run, so the throttle still limits normal replication for those replicas.
Fix
Run --verify with the same JSON file until it reports that throttles were cleared. Check kafka-configs.sh --describe --entity-type brokers for leftover throttle settings.
A replay that slows the live consumers
Symptom
A team re-reads a month of history. During the replay, produce latency and live consumers' fetch latency rise across unrelated topics.
Cause
Old segments are read from disk and fill the page cache, evicting the recent data that live consumers are normally served from. Disk reads now compete with writes.
Fix
Give replay clients a consumer_byte_rate quota, run them off-peak, and size memory for the lag you intend to support.

How to troubleshoot

Disk usage on brokers is climbing and you need to know which topics own it

Check

bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe --broker-list 1,2,3,4
How to read it
Lists each partition replica with its size in bytes per log directory. Sum by topic. The largest topics, multiplied by their retention, are where a change has an effect.
You need to know whether a reassignment is still running

Check

bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file expand.json --verify; metric kafka.server:type=ReplicaManager,name=ReassigningPartitions
How to read it
The tool lists each partition as completed or still in progress. The metric is non-zero on brokers leading partitions that are being moved.
Brokers respond slowly under load

Check

kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent and kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent
How to read it
Both are between 0 and 1 and should ideally stay above 0.3. Low request handler idle points at CPU or slow disk I/O in request processing; low network processor idle points at network threads, often with TLS.
One broker carries much more traffic than the others

Check

kafka.server:type=ReplicaManager,name=LeaderCount and name=PartitionCount per broker; kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
How to read it
A high leader count relative to partition count means leadership is uneven, typically after restarts. A high partition count means replicas are uneven and need reassignment.
After adding capacity, replication seems unhealthy

Check

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions; bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions
How to read it
Should be 0 in steady state. During a reassignment some partitions show extra replicas catching up. Persistent under-replication afterwards suggests a leftover throttle or an overloaded broker.
Tiered storage is on and local disks keep growing

Check

kafka.server:type=BrokerTopicMetrics,name=RemoteCopyLagBytes,topic=…
How to read it
Bytes in closed segments not yet copied to the remote tier. A growing value means uploads are failing or too slow, and local segments cannot be deleted until they are uploaded.

Key takeaways

  • Disk = byte rate as stored × retention × replication factor, divided by the fill level you allow. Do it per topic and add up.
  • Every produced byte crosses the network once in, once per follower, and once per consumer group. Size for the peak.
  • Kafka relies on the page cache: caught-up consumers are served from memory, lagging and replaying consumers from disk.
  • TLS and forced recompression are the avoidable or unavoidable CPU costs; watch the request handler and network processor idle ratios.
  • A new broker holds nothing until partitions are reassigned to it. Reassign with a throttle above the write rate, and clear it with --verify.
  • Scaling a consumer group is capped by partitions and by its dependencies; a new group is a full extra copy of outbound traffic.
  • Tiered storage moves old segments off broker disks. It is off by default, needs a storage plugin Kafka does not ship, and does not support compacted topics.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A new cluster will store an average of 12 MB/s as written by producers after compression. Retention is 5 days on every topic, the replication factor is 3, and the team does not want any disk above 60% full. They plan to run 6 brokers.

About how much disk does each broker need?

Scenario 2 of 3

A 5-broker cluster has disks at 82%. The team adds 3 brokers on Monday. On Wednesday the original five are at 83% and the new three show almost no disk use and no network traffic. No topics were created in between.

What explains this, and what is the next step?

Scenario 3 of 3

During a cluster expansion a reassignment was started with --throttle 20000000 (20 MB/s). After 18 hours, --verify still reports the partitions of one large topic as in progress and the new replicas' lag is flat. That topic's BytesInPerSec peaks at 35 MB/s during the day. Other, smaller topics in the same plan completed within hours.

What should the operator do?

References

References: Apache Kafka 4.4: Hardware and OS (memory estimate, disks, file descriptors, flush behaviour) ↗ · Apache Kafka 4.4: Basic Kafka operations (expanding your cluster, partition reassignment, limiting bandwidth during data migration) ↗ · Apache Kafka 4.4: Design (persistence, page cache, efficiency, zero-copy and TLS, batch compression) ↗ · Apache Kafka 4.4: Monitoring (broker byte rates, idle ratios, reassignment and tiered storage metrics) ↗ · Apache Kafka 4.4: Tiered storage (overview, configuration, limitations) ↗ · Apache Kafka 4.4: Broker configs (retention, segments, threads, replication throttles) ↗ · Apache Kafka 4.4: Topic configs (retention.ms, retention.bytes, segment.bytes, remote.storage.enable, local.retention.ms) ↗ · KIP-405: Kafka Tiered Storage ↗ · Report a problem