Path 8 · lesson 1 of 5
Throughput and latency
What sets each of them in a Kafka pipeline, why raising one usually costs the other, where the milliseconds go end to end, and how to measure with percentiles.
What you will learn
- Separate throughput from latency and name the stage that limits each one in a given pipeline
- Break the journey of one record into stages and say which setting or resource owns each stage
- Work out a consumer group's throughput ceiling and its utilisation from a processing time and a produce rate
- Explain why batching raises throughput and when it raises or lowers latency
- Run the performance tools shipped with Kafka and read their output
- Report latency as percentiles and explain what an average hides
Before you start
The concept
Throughput is how much work the pipeline finishes per second: records per second or bytes per second. Latency is how long one record takes. For an event pipeline the latency that matters is end to end: from the moment the application calls send() to the moment a consumer has finished processing that record.
They are different questions with different answers. Throughput is set by the slowest stage of the chain, because no stage can pass on more than it receives or handle more than its own limit. Latency is the sum of every stage the record passes through, including the time it spends waiting in a queue in front of a stage.
Throughput
- Records or bytes completed per second
- Limited by one stage: the bottleneck
- Raised by doing more per request (batching), by parallelism (partitions and consumers) or by making the bottleneck faster
- Speeding up a stage that is not the bottleneck changes nothing
Latency
- Time for one record, send to processed
- The sum of all stages plus all waiting
- Lowered by removing waits: shorter linger, less queueing, faster processing
- Dominated by queueing once any stage runs close to its limit
In most business pipelines the brokers are not the slow part. A consumer that makes a database call per record does a few hundred records per second per thread, while a broker moves many thousands of records in the same time. That is why this path starts with finding the bottleneck, not with broker settings.
Find the stage that sets the ceiling
Simulation · not a real clusterSix consumers at 10 ms per record can do 600 records per second, and 500 arrive. Change one input at a time and watch which limit the model names.
Why it exists
The two goals pull against each other because the cheapest way to raise throughput is to wait and group. Every network request, every append to the log and every database commit has a fixed cost that does not depend on how many records it carries. Send a thousand records in one request and that cost is paid once. The Kafka design documentation names the two inefficiencies it was built to avoid: too many small I/O operations and excessive byte copying. Batching is the answer to the first.
Grouping needs records to group, and collecting them takes time. A producer that waits up to linger.ms for more records makes the first record of the batch wait. A consumer fetch that waits for fetch.min.bytes holds back data that is already there. A listener that receives 500 records per poll and processes them one after another makes the last record wait for the 499 before it.
The second reason is utilisation. A stage that is busy 50% of the time usually finds records waiting only briefly. A stage that is busy 95% of the time has a queue in front of it most of the time, and a short burst takes a long while to clear. Running a pipeline at its maximum throughput therefore gives the worst latency it can have while still keeping up.
| Utilisation | Time in the stage, 10 ms of work | What it means |
|---|---|---|
| 50% | 10 / (1 − 0.50) = 20 ms | Half of the time is waiting |
| 80% | 10 / (1 − 0.80) = 50 ms | Waiting is four times the work |
| 90% | 10 / (1 − 0.90) = 100 ms | A common place to stop adding load |
| 95% | 10 / (1 − 0.95) = 200 ms | Small bursts become visible delays |
| 99% | 10 / (1 − 0.99) = 1,000 ms | Any burst becomes a backlog |
How it works inside
Where the time goes, stage by stage
| Stage | What happens | What sets its duration | How to see it |
|---|---|---|---|
| 1. Producer buffer | The record is serialised, assigned a partition and added to that partition's open batch | linger.ms and batch.size: the batch is sent when it is full or the linger time has passed | Producer metric record-queue-time-avg / -max |
| 2. Produce request | The sender thread sends the batch to the partition leader | Network round trip, and broker queueing if its threads are busy | Producer request-latency-avg; broker RequestQueueTimeMs |
| 3. Leader append | The leader appends the batch to the log, which means writing into the operating system's page cache | Disk and page cache behaviour; Kafka does not fsync per request by default | Broker LocalTimeMs, request=Produce |
| 4. Replication | Followers fetch the batch. With acks=all the leader answers only after every in-sync replica has it | Follower fetch round trip; a slow in-sync follower slows every acks=all write | Broker RemoteTimeMs, request=Produce |
| 5. Consumer fetch | The consumer's fetch request is answered once fetch.min.bytes is available or fetch.max.wait.ms has passed | With the default of 1 byte: answered as soon as data is readable | Consumer fetch-latency-avg |
| 6. Waiting in the poll batch | poll() hands over up to max.poll.records; records are processed in order | Position in the batch × processing time, when there is a backlog | Record age measured in the listener |
| 7. Processing | Your code: database write, HTTP call, business logic | Almost always the largest stage | A timer around the handler |
| 8. Commit | The position is committed; affects restart, not the record's own latency | One request per commit | Consumer commit-latency-avg |
kafka.network:type=RequestMetrics,name=…,request={Produce|FetchConsumer|FetchFollower}.A latency budget for one record
Write the stages down with numbers before tuning anything. The table is an example with assumed values for a pipeline that is keeping up, with brokers in one region. Your numbers come from the metrics in the previous table.
| Stage | Assumed time | Running total |
|---|---|---|
Wait in the producer batch (linger.ms 5, light traffic) | up to 5 ms | 5 ms |
| Produce request to the leader | 1 ms | 6 ms |
| Leader append to the page cache | 1 ms | 7 ms |
| Followers fetch, high watermark advances | 2 ms | 9 ms |
| Fetch response to the consumer (long poll already waiting) | 1 ms | 10 ms |
| Processing with one database write | 10 ms | 20 ms |
Now assume the same consumer has fallen behind and each poll() returns the full 500 records. The record at the end of that batch waits for 499 others: 499 × 10 ms = 4,990 ms before its own 10 ms of work. Nothing in the first five rows changed. This is why latency problems in Kafka consumers are nearly always queueing behind slow processing, and why lag (the queue) is the first thing to look at.
Why the broker is rarely the bottleneck
- Sequential I/O. A partition is an append-only file. The design documentation points out that linear reads and writes are the most predictable disk pattern and are heavily optimised by the operating system.
- Page cache. Writes go to the operating system's cache and are flushed in the background; reads of recent data are served from that cache. On a cluster whose consumers are mostly caught up, the documentation says you will see no read activity on the disks at all.
- Zero-copy. The on-disk format is the same as the wire format, so the broker can hand log data to the socket with
sendfilewithout copying it through the application. The documentation notes that this path is not used when TLS is enabled, because encryption happens in user space. - End-to-end batches. A batch compressed by the producer is stored compressed and sent to consumers compressed; the broker decompresses it only to validate it.
Percentiles, not averages
An average hides the cases users complain about. Take 1,000 records where 980 take 10 ms and 20 take 2,000 ms. The average is (980 × 10 + 20 × 2,000) / 1,000 = 49.8 ms, which describes none of them. The median (p50) is 10 ms and the 99th percentile is 2,000 ms. One record in fifty is 200 times slower than typical, and only the percentiles show it.
Report p50 for the typical case, p99 for the experience of the unlucky and the maximum for the worst case. Collect them as a histogram so they can be combined across instances: averaging the p99 of ten instances does not give the p99 of the service.
Architecture
Drawing diagram…
Flowchart of one record's path: producer batch, produce request, broker request queue, leader append, follower fetch, high watermark, then consumer fetch, poll batch, processing and offset commit. Labels show which settings control the waits.
flowchart LR APP["Order Service send"] --> BUF["Producer batch per partition"] BUF -->|"linger.ms or batch.size"| NET1["Produce request"] NET1 --> RQ["Broker request queue"] RQ --> APPEND["Leader append to page cache"] APPEND --> REPL["Followers fetch"] REPL --> HW["High watermark advances"] HW -->|"acks=all reply"| APP HW --> FETCH["Consumer fetch long poll"] FETCH -->|"fetch.min.bytes or fetch.max.wait.ms"| POLL["Poll batch up to max.poll.records"] POLL --> WORK["Processing and database write"] WORK --> COMMIT["Offset commit"]
- The left half is the produce path. Two waits are deliberate: the batch waits for
linger.msorbatch.size, and withacks=allthe reply waits until the in-sync followers have fetched the batch. - The right half is the consume path. The fetch is a long poll: the consumer asks, and the broker answers when there is enough data or the wait time is up.
- The high watermark box sits on both paths. It releases the producer's acknowledgement and it is also the point from which consumers may read.
- Processing is drawn the same size as the others, but in a typical service it is several times longer than everything before it together.
Message flow
Drawing diagram…
Sequence diagram: the producer batches a record and sends it to the leader, the follower fetches it, the high watermark advances, the leader acknowledges, the consumer's fetch returns the record, the consumer writes to the database and commits the offset.
sequenceDiagram participant P as Producer participant L as Leader broker participant F as Follower broker participant C as Consumer participant D as Database P->>P: send adds record to batch Note over P: wait up to linger.ms P->>L: Produce request with batch L->>L: append to log in page cache F->>L: Fetch as follower L-->>F: batch F->>L: next Fetch confirms position Note over L: high watermark advances L-->>P: ack for acks=all C->>L: Fetch long poll L-->>C: records up to high watermark C->>D: process record and write D-->>C: ok C->>L: commit offset
- The follower learns nothing by push. It fetches, and its next fetch tells the leader how far it has got. That is the moment the leader can advance the high watermark.
- The acknowledgement to the producer and the visibility to the consumer both depend on the high watermark, so a slow in-sync follower delays both.
- The consumer's fetch is normally already waiting at the broker when the record arrives, which is why the fetch stage is short when
fetch.min.bytesis 1. - The commit is after processing here (at-least-once). It adds a request but does not delay the record that was just processed.
In a real system
A typical order system publishes OrderPlaced events to a topic orders.events, keyed by order id, with 12 partitions and replication factor 3. The Payment Service reads it in the group payment-service and makes one database write per event.
Worked example: is this pipeline fast enough?
| Quantity | Assumption or arithmetic | Result |
|---|---|---|
| Peak produce rate | Assumed from last year's busiest hour | 800 records/s |
| Record size | Assumed, JSON before compression | 1 KB |
| Bytes in at peak | 800 × 1 KB | 0.8 MB/s |
| Processing time per record | Assumed, measured around the handler | 10 ms |
| One consumer thread | 1,000 ms / 10 ms | 100 records/s |
| Group ceiling | 12 partitions × 1 thread × 100 | 1,200 records/s |
| Utilisation at peak | 800 / 1,200 | 67% |
| Headroom | 1,200 − 800 | 400 records/s |
| Catch-up after a 5 minute stop | Backlog 800 × 300 = 240,000; 240,000 / 400 | 600 s = 10 minutes |
Two decisions follow. The throughput target is met with a third of the capacity spare, which is what keeps latency low and lets the group recover from a pause. And any tuning effort belongs in the 10 ms of processing, since that is where both the ceiling and most of the latency come from. Producer and broker settings would be the wrong place to start.
Java and Spring Boot
Measure before changing anything. The first file uses the performance tools that ship in Kafka's bin directory to find what the cluster and a bare client can do. The second measures the number that matters in production: how old each record is when your listener gets it, as a histogram.
# 1. Producer side: 1,000,000 records of 1 KB at a fixed 2,000 records/s.# A fixed rate shows latency at YOUR load. --throughput -1 removes the limit and shows the ceiling.bin/kafka-producer-perf-test.sh \ --bootstrap-server localhost:9092 \ --topic perf-orders \ --num-records 1000000 \ --record-size 1024 \ --throughput 2000 \ --command-property acks=all linger.ms=5 compression.type=lz4# 2. Consumer side: read the same records with no processing at all.bin/kafka-consumer-perf-test.sh \ --bootstrap-server localhost:9092 \ --topic perf-orders \ --group perf-read-1 \ --num-records 1000000 \ --show-detailed-stats# 3. End to end: one record at a time, produce then consume, with percentiles.bin/kafka-e2e-latency.sh \ --bootstrap-server localhost:9092 \ --topic perf-e2e \ --num-records 10000 \ --producer-acks all \ --record-size 1024Why it is written this way
- Use a test topic with the same partition count, replication factor and
min.insync.replicasas the real one. A single-replica topic measures a different system. - The producer tool's last line reports records/sec, MB/sec, average and maximum latency, and the 50th, 95th, 99th and 99.9th percentiles. Its latency is from
send()to the acknowledgement, so it includes the time spent waiting in the batch. - Run it twice: once with
--throughput -1to find the ceiling, then at your expected rate to read the latency you would really get. Latency at the ceiling is the worst case, not the normal one. - The consumer tool does no work per record, so its result is an upper bound on fetching, not a prediction for your service. Compare it with your processing rate to see how far apart they are.
- Option names changed over time. In 4.4 the producer tool takes
--bootstrap-serverand--command-property; older releases use--producer-props bootstrap.servers=…. The consumer tool's--num-recordswas--messages. The end-to-end tool took positional arguments in older releases. Run a tool with--helpon your version. - Real payloads compress differently from the tool's generated bytes. Use
--payload-filewith sample events when you compare compression codecs.
package com.example.payment;import java.time.Duration;import io.micrometer.core.instrument.MeterRegistry;import io.micrometer.core.instrument.Timer;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderEventListener { private final PaymentService payments; private final Timer recordAge; private final Timer processing; public OrderEventListener(PaymentService payments, MeterRegistry registry) { this.payments = payments; this.recordAge = Timer.builder("orders.events.record.age") .description("Time from the record timestamp to the start of processing") .publishPercentileHistogram() .register(registry); this.processing = Timer.builder("orders.events.processing") .description("Time spent in the payment handler") .publishPercentileHistogram() .register(registry); } @KafkaListener(id = "payment-service", topics = "orders.events") public void onOrderEvent(ConsumerRecord<String, OrderPlaced> record) { long ageMs = Math.max(0, System.currentTimeMillis() - record.timestamp()); recordAge.record(Duration.ofMillis(ageMs)); processing.record(() -> payments.handle(record.value())); }}Why it is written this way
record.timestamp()is set by the producer when the record is created (the topic defaultmessage.timestamp.typeisCreateTime), sonow − timestampcovers stages 1 to 6: batching, the produce request, replication, the fetch and the wait in the poll batch.- Two timers, because they answer different questions.
record.agerising whileprocessingstays flat means records are queueing: the group is near its ceiling or one partition is hot. publishPercentileHistogram()exports histogram buckets, so the monitoring system can compute p50, p99 and p99.9 across all instances. Percentiles computed inside each instance cannot be combined afterwards.- The age compares two clocks: the producer's and the consumer's. Keep hosts time-synchronised, and clamp at zero as here so a small skew does not produce a negative duration.
- A replay of old records shows a very large age while the consumer is healthy. Read this metric together with lag.
package com.example.payment;import java.math.BigDecimal;import java.time.Instant;public record OrderPlaced(String orderId, String customerId, BigDecimal total, Instant placedAt) {}Why it is written this way
- A record is enough for an event: immutable, with value equality, and Jackson can read and write it without extra annotations.
Configuration
These are the settings that put a deliberate wait or a size limit on the path. Defaults are from the Kafka 4.4 references. The next lessons in this path go through how to change them; here the point is to know which stage each one belongs to.
linger.ms (producer)Default 5
- What it does
- Upper bound on how long a record waits in its partition's batch for more records before the batch is sent. A full batch is sent at once.
- When to change it
- The default was 0 before Kafka 4.0, and Spring Boot 3.5 manages kafka-clients 3.9, so set it explicitly. Raise it for volume streams that can accept the delay; keep it low where a user is waiting.
batch.size (producer)Default 16384
- What it does
- Upper bound in bytes of one batch for one partition.
- When to change it
- Matters only when the per-partition rate fills a batch before linger expires. See the producer tuning lesson for the arithmetic.
acks (producer)Default all
- What it does
- How many acknowledgements the leader needs before it answers.
allwaits for every in-sync replica. - When to change it
- Leave it at
allfor business events. Lowering it shortens stage 4 for the producer only, gives up durability, and switches off idempotence if idempotence was not set explicitly.
fetch.min.bytes (consumer)Default 1
- What it does
- The broker answers a fetch as soon as this many bytes are available, or when
fetch.max.wait.mshas passed. - When to change it
- Keep 1 for low latency. Raise it for a high-volume stream where fewer, larger fetches are worth a wait.
fetch.max.wait.ms (consumer)Default 500
- What it does
- The longest the broker holds a fetch that has not reached
fetch.min.bytes. - When to change it
- It becomes the worst-case added latency once
fetch.min.bytesis above 1. Lower it together with raisingfetch.min.bytesif 500 ms is too long.
max.poll.records (consumer)Default 500
- What it does
- Maximum number of records one
poll()returns. It does not change how the consumer fetches; fetched records are cached and handed out in portions. - When to change it
- With slow processing, a smaller value shortens the wait inside a batch and keeps the time between polls well under
max.poll.interval.ms(default 300000).
replica.lag.time.max.ms (broker)Default 30000
- What it does
- A follower that has not caught up to the leader's log end for this long is removed from the in-sync set.
- When to change it
- Explains a pattern in produce latency: a follower that is slow but still in sync delays every
acks=allwrite until it catches up or is removed.
Hands-on exercise
Trade latency for throughput on purpose
Use the producer settings simulator to see what each batching choice does to request count, throughput and delay, and decide a setting for two different streams.
Steps · 0 of 6 done
Batching: fewer requests against added delay
Simulation · not a real clusterEvery outcome here is derived from the documented behaviour of the producer, not measured. Move one setting at a time.
What you should see
A longer linger gives fewer, larger requests and a higher relative throughput, and adds at most the linger time to a record's delay. In the model (1 KB records every 2 ms) a 16 KB batch fills after 30 ms, so at a linger of 50 the batch size is the limit and raising it gives larger batches, while at a linger of 5 the same change does nothing. acks=1 is rejected while idempotence is on, and once idempotence is off the panels warn that an acknowledged record can be lost and that retries can duplicate: a large price for a small latency gain. For checkout you should end at a small linger with acks=all; for the clickstream, a long linger, larger batches and compression.
What can go wrong
Tuning the producer when the consumer is the bottleneck
- Symptom
- Linger, batch size and compression have been changed several times. Producer metrics improved. End-to-end delay and lag are the same as before.
- Cause
- Throughput is set by the slowest stage. In this pipeline that is processing in the consumer, and nothing upstream of it was the limit.
- Fix
- Write the latency budget first. Compare the produce rate with the group's ceiling (threads × 1,000 / processing ms). Work on the stage with the largest number.
Sizing for the average rate
- Symptom
- The group keeps up most of the day and falls behind every evening, taking hours to recover.
- Cause
- Capacity was set from the daily average. At the peak the group runs above 100% utilisation and builds a backlog it can only clear with the small surplus it has off-peak.
- Fix
- Size for the peak rate with headroom (for example 60 to 70% utilisation at peak), and calculate the catch-up time for a realistic outage.
Benchmarking at maximum throughput and quoting that latency
- Symptom
- A load test reports p99 latency of several seconds and the team concludes Kafka is slow.
- Cause
- With
--throughput -1the producer sends as fast as it can, so records queue in the client buffer. The latency measured is the queueing delay of an overloaded client. - Fix
- Use the unlimited run to find the ceiling only. Measure latency at the rate you expect in production, and again at perhaps 1.5 times that rate.
Lowering acks to cut latency
- Symptom
- Produce latency drops a little. Weeks later a broker fails and acknowledged orders are missing.
- Cause
acks=1answers before followers have the record. Consumers still wait for the high watermark, so end-to-end latency barely moved, while durability was given up.- Fix
- Keep
acks=allwithmin.insync.replicas=2on a replication factor of 3. If replication is slow, find the slow follower throughRemoteTimeMsand under-replicated partitions.
Dashboards built on averages
- Symptom
- Average latency is flat at 30 ms while support tickets report confirmations arriving after a minute.
- Cause
- A small share of very slow records hardly moves an average. The slow records usually belong to one partition or one customer.
- Fix
- Export histograms and plot p50, p99 and the maximum. Break lag down per partition.
How to troubleshoot
Producer latency is high, and it is unclear whether the client, the network or the broker is responsible
Check
Producer metrics `record-queue-time-avg` and `request-latency-avg` (kafka.producer:type=producer-metrics)
- How to read it
- A high queue time with a normal request latency means records wait in the client: long linger, or the sender cannot keep up. A high request latency points at the network or the broker; continue with the broker's request metrics.
Produce requests are slow on the broker
Check
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce and its parts RequestQueueTimeMs, LocalTimeMs, RemoteTimeMs, ResponseQueueTimeMs, ResponseSendTimeMs
- How to read it
- High
RequestQueueTimeMs: request handler threads are busy (checkRequestHandlerAvgIdlePercent, which the docs say should ideally stay above 0.3). HighLocalTimeMs: the disk or page cache. HighRemoteTimeMs: waiting for followers underacks=all.
Records are old by the time the listener sees them
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
- How to read it
- Growing
LAGon every partition means the group is above its ceiling. Lag on one partition means a hot or stuck partition. No lag with old records means the delay is on the produce side or the timestamps are old (a replay).
The consumer adds a steady delay of up to half a second at low traffic
Check
Consumer settings `fetch.min.bytes` and `fetch.max.wait.ms`; consumer metric `fetch-latency-avg`
- How to read it
- If
fetch.min.byteswas raised, the broker holds each fetch until that much data exists orfetch.max.wait.ms(default 500) passes. At low traffic the timeout is what releases the data.
You need a baseline for what the cluster can do
Check
bin/kafka-producer-perf-test.sh --bootstrap-server localhost:9092 --topic perf-orders --num-records 1000000 --record-size 1024 --throughput -1 --command-property acks=all
- How to read it
- The final line gives records/sec and MB/sec at the ceiling for this client and topic. If your application's rate is far below it, the limit is in the application.
Key takeaways
- Throughput is set by the slowest stage; latency is the sum of all stages plus the time spent queueing in front of them.
- Batching trades a bounded wait for fewer, larger requests. Under load it often lowers latency as well, which is why
linger.msdefaults to 5 since Kafka 4.0. - A consumer reads only up to the high watermark, so replication time is on every record's path whatever
acksis. - Latency climbs steeply as a stage approaches full utilisation. Size for the peak with headroom and know your catch-up time.
- In business pipelines the consumer's processing is usually the bottleneck. Calculate threads × 1,000 / processing ms before touching broker or producer settings.
- Measure with the tools Kafka ships and with a record-age histogram, and report p50, p99 and the maximum.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A notification pipeline produces 900 records per second at peak. The topic has 6 partitions and the group has 6 consumer threads, each taking 8 ms per record. Lag grows every evening and clears overnight. A colleague proposes raising the producer's linger.ms from 5 to 50 to increase throughput.
What do the numbers say about that proposal?
Scenario 2 of 3
A team measures end-to-end latency for order events: p50 is 22 ms and the average is 31 ms, both within target. The p99 is 4.8 seconds. The broker's produce and fetch request times are a few milliseconds at every percentile. Lag per partition is under 20 records on eleven partitions and several thousand on one.
Where should they look first?
Scenario 3 of 3
A payments producer uses acks=all on a topic with replication factor 3 and min.insync.replicas 2. Produce p99 is 40 ms against a 25 ms target. The broker metric RemoteTimeMs for Produce is 35 ms at p99, LocalTimeMs is 1 ms, and one broker shows a much higher fetch-follower latency than the other two.
Which action addresses the cause without giving up a guarantee?
References
References: Apache Kafka 4.4: Design (persistence, efficiency, zero-copy, end-to-end batch compression) ↗ · Apache Kafka 4.4: Producer configs (linger.ms, batch.size, acks) ↗ · Apache Kafka 4.4: Consumer configs (fetch.min.bytes, fetch.max.wait.ms, max.poll.records) ↗ · Apache Kafka 4.4: Broker configs (min.insync.replicas, replica.lag.time.max.ms) ↗ · Apache Kafka 4.4: Monitoring (request metrics, producer and consumer metrics) ↗ · Apache Kafka source, 4.4 branch: ProducerPerformance (kafka-producer-perf-test.sh options and output) ↗ · Apache Kafka source, 4.4 branch: EndToEndLatency (kafka-e2e-latency.sh options) ↗ · Apache Kafka 3.9: Producer configs (linger.ms default 0 in the client Spring Boot 3.5 manages) ↗ · Report a problem