Skip to content

Path 8 · lesson 1 of 5

Intermediate24 minApache Kafka 4.4 (configuration, monitoring and design pages; perf tools read from the 4.4 source) · Spring Boot 3.5 with Spring for Apache Kafka 3.3, which manages kafka-clients 3.9

Throughput and latency

What sets each of them in a Kafka pipeline, why raising one usually costs the other, where the milliseconds go end to end, and how to measure with percentiles.

What you will learn

  • Separate throughput from latency and name the stage that limits each one in a given pipeline
  • Break the journey of one record into stages and say which setting or resource owns each stage
  • Work out a consumer group's throughput ceiling and its utilisation from a processing time and a produce rate
  • Explain why batching raises throughput and when it raises or lowers latency
  • Run the performance tools shipped with Kafka and read their output
  • Report latency as percentiles and explain what an average hides

Before you start

The concept

Throughput is how much work the pipeline finishes per second: records per second or bytes per second. Latency is how long one record takes. For an event pipeline the latency that matters is end to end: from the moment the application calls send() to the moment a consumer has finished processing that record.

They are different questions with different answers. Throughput is set by the slowest stage of the chain, because no stage can pass on more than it receives or handle more than its own limit. Latency is the sum of every stage the record passes through, including the time it spends waiting in a queue in front of a stage.

Throughput

  • Records or bytes completed per second
  • Limited by one stage: the bottleneck
  • Raised by doing more per request (batching), by parallelism (partitions and consumers) or by making the bottleneck faster
  • Speeding up a stage that is not the bottleneck changes nothing

Latency

  • Time for one record, send to processed
  • The sum of all stages plus all waiting
  • Lowered by removing waits: shorter linger, less queueing, faster processing
  • Dominated by queueing once any stage runs close to its limit

In most business pipelines the brokers are not the slow part. A consumer that makes a database call per record does a few hundred records per second per thread, while a broker moves many thousands of records in the same time. That is why this path starts with finding the bottleneck, not with broker settings.

Find the stage that sets the ceiling

Simulation · not a real cluster

Six consumers at 10 ms per record can do 600 records per second, and 500 arrive. Change one input at a time and watch which limit the model names.

Why it exists

The two goals pull against each other because the cheapest way to raise throughput is to wait and group. Every network request, every append to the log and every database commit has a fixed cost that does not depend on how many records it carries. Send a thousand records in one request and that cost is paid once. The Kafka design documentation names the two inefficiencies it was built to avoid: too many small I/O operations and excessive byte copying. Batching is the answer to the first.

Grouping needs records to group, and collecting them takes time. A producer that waits up to linger.ms for more records makes the first record of the batch wait. A consumer fetch that waits for fetch.min.bytes holds back data that is already there. A listener that receives 500 records per poll and processes them one after another makes the last record wait for the 499 before it.

The second reason is utilisation. A stage that is busy 50% of the time usually finds records waiting only briefly. A stage that is busy 95% of the time has a queue in front of it most of the time, and a short burst takes a long while to clear. Running a pipeline at its maximum throughput therefore gives the worst latency it can have while still keeping up.

UtilisationTime in the stage, 10 ms of workWhat it means
50%10 / (1 − 0.50) = 20 msHalf of the time is waiting
80%10 / (1 − 0.80) = 50 msWaiting is four times the work
90%10 / (1 − 0.90) = 100 msA common place to stop adding load
95%10 / (1 − 0.95) = 200 msSmall bursts become visible delays
99%10 / (1 − 0.99) = 1,000 msAny burst becomes a backlog
A model, not a measurement: the simplest queueing formula (random arrivals, one worker), average time = work time / (1 − utilisation). Real consumers differ in the numbers, not in the shape of the curve.

How it works inside

Where the time goes, stage by stage

StageWhat happensWhat sets its durationHow to see it
1. Producer bufferThe record is serialised, assigned a partition and added to that partition's open batchlinger.ms and batch.size: the batch is sent when it is full or the linger time has passedProducer metric record-queue-time-avg / -max
2. Produce requestThe sender thread sends the batch to the partition leaderNetwork round trip, and broker queueing if its threads are busyProducer request-latency-avg; broker RequestQueueTimeMs
3. Leader appendThe leader appends the batch to the log, which means writing into the operating system's page cacheDisk and page cache behaviour; Kafka does not fsync per request by defaultBroker LocalTimeMs, request=Produce
4. ReplicationFollowers fetch the batch. With acks=all the leader answers only after every in-sync replica has itFollower fetch round trip; a slow in-sync follower slows every acks=all writeBroker RemoteTimeMs, request=Produce
5. Consumer fetchThe consumer's fetch request is answered once fetch.min.bytes is available or fetch.max.wait.ms has passedWith the default of 1 byte: answered as soon as data is readableConsumer fetch-latency-avg
6. Waiting in the poll batchpoll() hands over up to max.poll.records; records are processed in orderPosition in the batch × processing time, when there is a backlogRecord age measured in the listener
7. ProcessingYour code: database write, HTTP call, business logicAlmost always the largest stageA timer around the handler
8. CommitThe position is committed; affects restart, not the record's own latencyOne request per commitConsumer commit-latency-avg
Metric names are from the Kafka 4.4 monitoring reference. Broker request metrics are kafka.network:type=RequestMetrics,name=…,request={Produce|FetchConsumer|FetchFollower}.

A latency budget for one record

Write the stages down with numbers before tuning anything. The table is an example with assumed values for a pipeline that is keeping up, with brokers in one region. Your numbers come from the metrics in the previous table.

StageAssumed timeRunning total
Wait in the producer batch (linger.ms 5, light traffic)up to 5 ms5 ms
Produce request to the leader1 ms6 ms
Leader append to the page cache1 ms7 ms
Followers fetch, high watermark advances2 ms9 ms
Fetch response to the consumer (long poll already waiting)1 ms10 ms
Processing with one database write10 ms20 ms
In this example processing is half of the total and the broker stages together are 4 ms. Halving linger saves 2.5 ms; halving the database call saves 5 ms.

Now assume the same consumer has fallen behind and each poll() returns the full 500 records. The record at the end of that batch waits for 499 others: 499 × 10 ms = 4,990 ms before its own 10 ms of work. Nothing in the first five rows changed. This is why latency problems in Kafka consumers are nearly always queueing behind slow processing, and why lag (the queue) is the first thing to look at.

Why the broker is rarely the bottleneck

  • Sequential I/O. A partition is an append-only file. The design documentation points out that linear reads and writes are the most predictable disk pattern and are heavily optimised by the operating system.
  • Page cache. Writes go to the operating system's cache and are flushed in the background; reads of recent data are served from that cache. On a cluster whose consumers are mostly caught up, the documentation says you will see no read activity on the disks at all.
  • Zero-copy. The on-disk format is the same as the wire format, so the broker can hand log data to the socket with sendfile without copying it through the application. The documentation notes that this path is not used when TLS is enabled, because encryption happens in user space.
  • End-to-end batches. A batch compressed by the producer is stored compressed and sent to consumers compressed; the broker decompresses it only to validate it.

Percentiles, not averages

An average hides the cases users complain about. Take 1,000 records where 980 take 10 ms and 20 take 2,000 ms. The average is (980 × 10 + 20 × 2,000) / 1,000 = 49.8 ms, which describes none of them. The median (p50) is 10 ms and the 99th percentile is 2,000 ms. One record in fifty is 200 times slower than typical, and only the percentiles show it.

Report p50 for the typical case, p99 for the experience of the unlucky and the maximum for the worst case. Collect them as a histogram so they can be combined across instances: averaging the p99 of ten instances does not give the p99 of the service.

Architecture

Drawing diagram…

Flowchart of one record's path: producer batch, produce request, broker request queue, leader append, follower fetch, high watermark, then consumer fetch, poll batch, processing and offset commit. Labels show which settings control the waits.

flowchart LR
  APP["Order Service send"] --> BUF["Producer batch per partition"]
  BUF -->|"linger.ms or batch.size"| NET1["Produce request"]
  NET1 --> RQ["Broker request queue"]
  RQ --> APPEND["Leader append to page cache"]
  APPEND --> REPL["Followers fetch"]
  REPL --> HW["High watermark advances"]
  HW -->|"acks=all reply"| APP
  HW --> FETCH["Consumer fetch long poll"]
  FETCH -->|"fetch.min.bytes or fetch.max.wait.ms"| POLL["Poll batch up to max.poll.records"]
  POLL --> WORK["Processing and database write"]
  WORK --> COMMIT["Offset commit"]
Every box adds latency. The slowest box sets throughput.
  • The left half is the produce path. Two waits are deliberate: the batch waits for linger.ms or batch.size, and with acks=all the reply waits until the in-sync followers have fetched the batch.
  • The right half is the consume path. The fetch is a long poll: the consumer asks, and the broker answers when there is enough data or the wait time is up.
  • The high watermark box sits on both paths. It releases the producer's acknowledgement and it is also the point from which consumers may read.
  • Processing is drawn the same size as the others, but in a typical service it is several times longer than everything before it together.

Message flow

Drawing diagram…

Sequence diagram: the producer batches a record and sends it to the leader, the follower fetches it, the high watermark advances, the leader acknowledges, the consumer's fetch returns the record, the consumer writes to the database and commits the offset.

sequenceDiagram
  participant P as Producer
  participant L as Leader broker
  participant F as Follower broker
  participant C as Consumer
  participant D as Database
  P->>P: send adds record to batch
  Note over P: wait up to linger.ms
  P->>L: Produce request with batch
  L->>L: append to log in page cache
  F->>L: Fetch as follower
  L-->>F: batch
  F->>L: next Fetch confirms position
  Note over L: high watermark advances
  L-->>P: ack for acks=all
  C->>L: Fetch long poll
  L-->>C: records up to high watermark
  C->>D: process record and write
  D-->>C: ok
  C->>L: commit offset
One record from send to commit, with the two waits on the produce side and the long poll on the consume side.
  1. The follower learns nothing by push. It fetches, and its next fetch tells the leader how far it has got. That is the moment the leader can advance the high watermark.
  2. The acknowledgement to the producer and the visibility to the consumer both depend on the high watermark, so a slow in-sync follower delays both.
  3. The consumer's fetch is normally already waiting at the broker when the record arrives, which is why the fetch stage is short when fetch.min.bytes is 1.
  4. The commit is after processing here (at-least-once). It adds a request but does not delay the record that was just processed.

In a real system

A typical order system publishes OrderPlaced events to a topic orders.events, keyed by order id, with 12 partitions and replication factor 3. The Payment Service reads it in the group payment-service and makes one database write per event.

Worked example: is this pipeline fast enough?

QuantityAssumption or arithmeticResult
Peak produce rateAssumed from last year's busiest hour800 records/s
Record sizeAssumed, JSON before compression1 KB
Bytes in at peak800 × 1 KB0.8 MB/s
Processing time per recordAssumed, measured around the handler10 ms
One consumer thread1,000 ms / 10 ms100 records/s
Group ceiling12 partitions × 1 thread × 1001,200 records/s
Utilisation at peak800 / 1,20067%
Headroom1,200 − 800400 records/s
Catch-up after a 5 minute stopBacklog 800 × 300 = 240,000; 240,000 / 400600 s = 10 minutes
In this example the bottleneck is the consumer, at 1,200 records per second. 0.8 MB/s is a small load for a broker.

Two decisions follow. The throughput target is met with a third of the capacity spare, which is what keeps latency low and lets the group recover from a pause. And any tuning effort belongs in the 10 ms of processing, since that is where both the ceiling and most of the latency come from. Producer and broker settings would be the wrong place to start.

Java and Spring Boot

Measure before changing anything. The first file uses the performance tools that ship in Kafka's bin directory to find what the cluster and a bare client can do. The second measures the number that matters in production: how old each record is when your listener gets it, as a histogram.

measure.shShell
# 1. Producer side: 1,000,000 records of 1 KB at a fixed 2,000 records/s.#    A fixed rate shows latency at YOUR load. --throughput -1 removes the limit and shows the ceiling.bin/kafka-producer-perf-test.sh \  --bootstrap-server localhost:9092 \  --topic perf-orders \  --num-records 1000000 \  --record-size 1024 \  --throughput 2000 \  --command-property acks=all linger.ms=5 compression.type=lz4# 2. Consumer side: read the same records with no processing at all.bin/kafka-consumer-perf-test.sh \  --bootstrap-server localhost:9092 \  --topic perf-orders \  --group perf-read-1 \  --num-records 1000000 \  --show-detailed-stats# 3. End to end: one record at a time, produce then consume, with percentiles.bin/kafka-e2e-latency.sh \  --bootstrap-server localhost:9092 \  --topic perf-e2e \  --num-records 10000 \  --producer-acks all \  --record-size 1024

Why it is written this way

  • Use a test topic with the same partition count, replication factor and min.insync.replicas as the real one. A single-replica topic measures a different system.
  • The producer tool's last line reports records/sec, MB/sec, average and maximum latency, and the 50th, 95th, 99th and 99.9th percentiles. Its latency is from send() to the acknowledgement, so it includes the time spent waiting in the batch.
  • Run it twice: once with --throughput -1 to find the ceiling, then at your expected rate to read the latency you would really get. Latency at the ceiling is the worst case, not the normal one.
  • The consumer tool does no work per record, so its result is an upper bound on fetching, not a prediction for your service. Compare it with your processing rate to see how far apart they are.
  • Option names changed over time. In 4.4 the producer tool takes --bootstrap-server and --command-property; older releases use --producer-props bootstrap.servers=…. The consumer tool's --num-records was --messages. The end-to-end tool took positional arguments in older releases. Run a tool with --help on your version.
  • Real payloads compress differently from the tool's generated bytes. Use --payload-file with sample events when you compare compression codecs.

Configuration

These are the settings that put a deliberate wait or a size limit on the path. Defaults are from the Kafka 4.4 references. The next lessons in this path go through how to change them; here the point is to know which stage each one belongs to.

linger.ms (producer)Default 5
What it does
Upper bound on how long a record waits in its partition's batch for more records before the batch is sent. A full batch is sent at once.
When to change it
The default was 0 before Kafka 4.0, and Spring Boot 3.5 manages kafka-clients 3.9, so set it explicitly. Raise it for volume streams that can accept the delay; keep it low where a user is waiting.
batch.size (producer)Default 16384
What it does
Upper bound in bytes of one batch for one partition.
When to change it
Matters only when the per-partition rate fills a batch before linger expires. See the producer tuning lesson for the arithmetic.
acks (producer)Default all
What it does
How many acknowledgements the leader needs before it answers. all waits for every in-sync replica.
When to change it
Leave it at all for business events. Lowering it shortens stage 4 for the producer only, gives up durability, and switches off idempotence if idempotence was not set explicitly.
fetch.min.bytes (consumer)Default 1
What it does
The broker answers a fetch as soon as this many bytes are available, or when fetch.max.wait.ms has passed.
When to change it
Keep 1 for low latency. Raise it for a high-volume stream where fewer, larger fetches are worth a wait.
fetch.max.wait.ms (consumer)Default 500
What it does
The longest the broker holds a fetch that has not reached fetch.min.bytes.
When to change it
It becomes the worst-case added latency once fetch.min.bytes is above 1. Lower it together with raising fetch.min.bytes if 500 ms is too long.
max.poll.records (consumer)Default 500
What it does
Maximum number of records one poll() returns. It does not change how the consumer fetches; fetched records are cached and handed out in portions.
When to change it
With slow processing, a smaller value shortens the wait inside a batch and keeps the time between polls well under max.poll.interval.ms (default 300000).
replica.lag.time.max.ms (broker)Default 30000
What it does
A follower that has not caught up to the leader's log end for this long is removed from the in-sync set.
When to change it
Explains a pattern in produce latency: a follower that is slow but still in sync delays every acks=all write until it catches up or is removed.

Hands-on exercise

Trade latency for throughput on purpose

Use the producer settings simulator to see what each batching choice does to request count, throughput and delay, and decide a setting for two different streams.

Steps · 0 of 6 done

Batching: fewer requests against added delay

Simulation · not a real cluster

Every outcome here is derived from the documented behaviour of the producer, not measured. Move one setting at a time.

Take it further in the lab: Consumer lag during high load

What can go wrong

Tuning the producer when the consumer is the bottleneck
Symptom
Linger, batch size and compression have been changed several times. Producer metrics improved. End-to-end delay and lag are the same as before.
Cause
Throughput is set by the slowest stage. In this pipeline that is processing in the consumer, and nothing upstream of it was the limit.
Fix
Write the latency budget first. Compare the produce rate with the group's ceiling (threads × 1,000 / processing ms). Work on the stage with the largest number.
Sizing for the average rate
Symptom
The group keeps up most of the day and falls behind every evening, taking hours to recover.
Cause
Capacity was set from the daily average. At the peak the group runs above 100% utilisation and builds a backlog it can only clear with the small surplus it has off-peak.
Fix
Size for the peak rate with headroom (for example 60 to 70% utilisation at peak), and calculate the catch-up time for a realistic outage.
Benchmarking at maximum throughput and quoting that latency
Symptom
A load test reports p99 latency of several seconds and the team concludes Kafka is slow.
Cause
With --throughput -1 the producer sends as fast as it can, so records queue in the client buffer. The latency measured is the queueing delay of an overloaded client.
Fix
Use the unlimited run to find the ceiling only. Measure latency at the rate you expect in production, and again at perhaps 1.5 times that rate.
Lowering acks to cut latency
Symptom
Produce latency drops a little. Weeks later a broker fails and acknowledged orders are missing.
Cause
acks=1 answers before followers have the record. Consumers still wait for the high watermark, so end-to-end latency barely moved, while durability was given up.
Fix
Keep acks=all with min.insync.replicas=2 on a replication factor of 3. If replication is slow, find the slow follower through RemoteTimeMs and under-replicated partitions.
Dashboards built on averages
Symptom
Average latency is flat at 30 ms while support tickets report confirmations arriving after a minute.
Cause
A small share of very slow records hardly moves an average. The slow records usually belong to one partition or one customer.
Fix
Export histograms and plot p50, p99 and the maximum. Break lag down per partition.

How to troubleshoot

Producer latency is high, and it is unclear whether the client, the network or the broker is responsible

Check

Producer metrics `record-queue-time-avg` and `request-latency-avg` (kafka.producer:type=producer-metrics)
How to read it
A high queue time with a normal request latency means records wait in the client: long linger, or the sender cannot keep up. A high request latency points at the network or the broker; continue with the broker's request metrics.
Produce requests are slow on the broker

Check

kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce and its parts RequestQueueTimeMs, LocalTimeMs, RemoteTimeMs, ResponseQueueTimeMs, ResponseSendTimeMs
How to read it
High RequestQueueTimeMs: request handler threads are busy (check RequestHandlerAvgIdlePercent, which the docs say should ideally stay above 0.3). High LocalTimeMs: the disk or page cache. High RemoteTimeMs: waiting for followers under acks=all.
Records are old by the time the listener sees them

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
Growing LAG on every partition means the group is above its ceiling. Lag on one partition means a hot or stuck partition. No lag with old records means the delay is on the produce side or the timestamps are old (a replay).
The consumer adds a steady delay of up to half a second at low traffic

Check

Consumer settings `fetch.min.bytes` and `fetch.max.wait.ms`; consumer metric `fetch-latency-avg`
How to read it
If fetch.min.bytes was raised, the broker holds each fetch until that much data exists or fetch.max.wait.ms (default 500) passes. At low traffic the timeout is what releases the data.
You need a baseline for what the cluster can do

Check

bin/kafka-producer-perf-test.sh --bootstrap-server localhost:9092 --topic perf-orders --num-records 1000000 --record-size 1024 --throughput -1 --command-property acks=all
How to read it
The final line gives records/sec and MB/sec at the ceiling for this client and topic. If your application's rate is far below it, the limit is in the application.

Key takeaways

  • Throughput is set by the slowest stage; latency is the sum of all stages plus the time spent queueing in front of them.
  • Batching trades a bounded wait for fewer, larger requests. Under load it often lowers latency as well, which is why linger.ms defaults to 5 since Kafka 4.0.
  • A consumer reads only up to the high watermark, so replication time is on every record's path whatever acks is.
  • Latency climbs steeply as a stage approaches full utilisation. Size for the peak with headroom and know your catch-up time.
  • In business pipelines the consumer's processing is usually the bottleneck. Calculate threads × 1,000 / processing ms before touching broker or producer settings.
  • Measure with the tools Kafka ships and with a record-age histogram, and report p50, p99 and the maximum.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A notification pipeline produces 900 records per second at peak. The topic has 6 partitions and the group has 6 consumer threads, each taking 8 ms per record. Lag grows every evening and clears overnight. A colleague proposes raising the producer's linger.ms from 5 to 50 to increase throughput.

What do the numbers say about that proposal?

Scenario 2 of 3

A team measures end-to-end latency for order events: p50 is 22 ms and the average is 31 ms, both within target. The p99 is 4.8 seconds. The broker's produce and fetch request times are a few milliseconds at every percentile. Lag per partition is under 20 records on eleven partitions and several thousand on one.

Where should they look first?

Scenario 3 of 3

A payments producer uses acks=all on a topic with replication factor 3 and min.insync.replicas 2. Produce p99 is 40 ms against a 25 ms target. The broker metric RemoteTimeMs for Produce is 35 ms at p99, LocalTimeMs is 1 ms, and one broker shows a much higher fetch-follower latency than the other two.

Which action addresses the cause without giving up a guarantee?

References

References: Apache Kafka 4.4: Design (persistence, efficiency, zero-copy, end-to-end batch compression) ↗ · Apache Kafka 4.4: Producer configs (linger.ms, batch.size, acks) ↗ · Apache Kafka 4.4: Consumer configs (fetch.min.bytes, fetch.max.wait.ms, max.poll.records) ↗ · Apache Kafka 4.4: Broker configs (min.insync.replicas, replica.lag.time.max.ms) ↗ · Apache Kafka 4.4: Monitoring (request metrics, producer and consumer metrics) ↗ · Apache Kafka source, 4.4 branch: ProducerPerformance (kafka-producer-perf-test.sh options and output) ↗ · Apache Kafka source, 4.4 branch: EndToEndLatency (kafka-e2e-latency.sh options) ↗ · Apache Kafka 3.9: Producer configs (linger.ms default 0 in the client Spring Boot 3.5 manages) ↗ · Report a problem