Path 8 · lesson 3 of 5
Tuning producers and consumer fetching
A procedure for tuning batching, compression, buffers, fetch sizes and large messages: calculate first, change one setting at a time, and measure.
What you will learn
- Calculate from a stream's per-partition byte rate whether linger.ms or batch.size limits its batches, before changing either
- Follow a tuning procedure: goal, baseline, one change, same load, compare, keep or revert
- Read the producer metrics that show batch size, queue time, compression ratio and buffer exhaustion
- Tune the consumer's fetch settings for a high-volume stream and say what each one costs in latency
- Trace a record that is too large through every size limit on its path and decide between raising limits and sending a reference
- Tell tuning settings apart from guarantee settings that must not be traded for speed
Before you start
The concept
Tuning is a procedure, not a list of recommended values. The same setting that doubles the throughput of a clickstream does nothing for an order stream, because the two differ in the one number that decides how batching behaves: bytes per second arriving for one partition in one producer instance.
The mechanics of batches, compression and the producer buffer are covered in Batching, linger.ms, compression and buffer.memory. This lesson is about using them: how to predict what a change will do, how to check that it did, and where to stop.
- State the goal as numbers. For example: carry 20,000 records per second with produce p99 under 150 ms. Without a target there is no point at which you are done.
- Fix the guarantees.
acks,enable.idempotence,min.insync.replicasand the replication factor are decided by what the data is worth. They are inputs to tuning, not part of it. - Record a baseline. Write down every setting explicitly and capture the metrics under a load you can repeat.
- Calculate. From the per-partition byte rate, work out what batches the current settings can produce and which setting is the limit.
- Change one setting. Run the same load for the same time. Compare the same metrics.
- Keep or revert. If the change did not move the target metric, undo it. Settings that do nothing still have to be explained to the next reader.
- Stop when the goal is met.
Which setting is the limit?
Simulation · not a real clusterThe picture assumes 1 KB records arriving every 2 ms for one partition. Raise linger.ms step by step and read the reason each batch was sent.
Why it exists
Producer and consumer settings interact, and several of them only matter in a certain range. batch.size has no effect while batches are sent by the linger timer. linger.ms has no effect once batches fill before the timer. Compression gains depend on how many similar records share a batch. A change made without knowing which range you are in is a guess, and guesses accumulate: a configuration ends up with ten non-default values, of which two do something.
Changing several settings together has a second cost. If throughput rises 30% after linger, batch size, compression and buffer size were all changed, you do not know which change earned it, which ones are costing latency or CPU for nothing, and what to undo when a problem appears later.
A third reason is version drift. linger.ms defaults to 5 in Kafka 4.x clients and to 0 in the 3.9 client that Spring Boot 3.5 manages. An application that relies on the default changes behaviour when the client library is upgraded. Tuned settings should be written down explicitly.
How it works inside
Step 1: which setting limits the batch?
A batch for a partition is sent when it reaches batch.size bytes or when linger.ms has passed since its first record, whichever comes first. So compare two times: the linger time, and the time the stream needs to fill a batch.
- Per-partition byte rate = records per second × record size / partitions / producer instances.
- Time to fill =
batch.size/ per-partition byte rate. - If time to fill is longer than
linger.ms, linger is the limit: batches hold (byte rate × linger) bytes, and raisingbatch.sizedoes nothing. - If time to fill is shorter than
linger.ms, batch size is the limit: batches are full, and raisinglinger.msdoes nothing.
| Order events | Clickstream | |
|---|---|---|
| Assumed rate and size | 1,200 records/s × 1 KB | 20,000 records/s × 300 B |
| Assumed partitions and producer instances | 24 partitions, 4 instances | 12 partitions, 2 instances |
| Per-partition rate per instance | 1,200 / 24 / 4 = 12.5 records/s = 12.5 KB/s | 20,000 / 12 / 2 = 833 records/s = 250 KB/s |
| Time to fill 16,384 bytes | 16,384 / 12,500 = 1.3 s | 16,384 / 250,000 = 66 ms |
Batch at linger.ms 5 | 12.5 × 0.005 = 0.06 records: one record per batch | 833 × 0.005 = 4 records, 1.25 KB |
Batch at linger.ms 50 | 0.6 records: still one per batch | 42 records, 12.5 KB |
Batch at linger.ms 100 | 1.25 records | Full at 16 KB after 66 ms: batch size is now the limit |
Batch at linger.ms 100, batch.size 65,536 | No change | 250 KB/s × 0.1 s = 25 KB, 83 records |
Two things move real batches away from this estimate. Under load the sender may be waiting for earlier requests, and the producer reference notes that this broker backpressure gives a higher effective linger, so batches grow by themselves when the producer is busy. And records without a key behave differently: the built-in partitioner stays on one partition until about batch.size bytes have been produced to it and then moves on, so batches fill regardless of the partition count.
Step 2: read the baseline
| Producer metric | What it tells you | What to compare it with |
|---|---|---|
batch-size-avg | Average bytes per batch actually sent (after compression) | batch.size. Far below it: linger is the limit. Close to it: batch size is the limit |
records-per-request-avg | Average records per request | 1 means batching is not happening |
record-queue-time-avg / -max | Time batches spent in the buffer before being sent | linger.ms. Well above it: the sender or the broker is the bottleneck |
request-latency-avg | Time from sending a request to its response | Rises with a slow broker or slow replication under acks=all |
compression-rate-avg | Compressed size divided by uncompressed size | 0.3 means the batch shrank to 30%. Close to 1 means compression is wasted CPU |
buffer-available-bytes, bufferpool-wait-ratio | Free buffer memory and the share of time send() waited for it | A wait ratio above 0 means the application produces faster than the producer can send |
produce-throttle-time-avg | Time the broker throttled this client because of a quota | Above 0: the limit is a quota, not a setting |
Step 3: change in this order
linger.ms, if the calculation says linger is the limit and the stream can accept the delay. This is usually the largest single gain for a busy keyed stream.batch.size, only once batches are full. The reference notes that a buffer of this size is allocated for each batch, so a very large value uses memory in proportion to the number of partitions being written.compression.type, once batches hold many records. Compression works on whole batches, so the ratio improves with batch size. Comparelz4andzstdon your own payloads withcompression-rate-avgand CPU.buffer.memory, only ifbufferpool-wait-ratiois above zero during normal operation and the cause is a burst the producer can absorb. If the broker is simply slower than the application, a larger buffer delays the failure and hides it.- Consumer fetch settings, if the consumer side of a high-volume stream is making many small fetches.
The consumer side
| Setting | Default | Effect | Cost |
|---|---|---|---|
fetch.min.bytes | 1 | The broker holds a fetch until this much data is available | Adds waiting time at low traffic, up to fetch.max.wait.ms |
fetch.max.wait.ms | 500 | Upper bound on that wait | Worst-case added latency when fetch.min.bytes is not reached |
max.partition.fetch.bytes | 1048576 | Upper bound on data per partition per fetch (a larger first batch is still returned) | Memory: up to this much per assigned partition, within fetch.max.bytes |
max.poll.records | 500 | How many fetched records one poll() hands to your code | None for fetching: the reference states it does not change the underlying fetch behaviour |
For the clickstream, one partition receives 20,000 × 300 B / 12 = 500 KB/s. With fetch.min.bytes at 65,536, a fetch for one partition is answered after 65,536 / 500,000 = 0.13 seconds, about 8 fetches per second instead of one for every small batch that arrives. The price is up to 130 ms of added delay, acceptable for a dashboard and not for a checkout. For the order stream at 50 KB/s per partition the same setting would hold every fetch for the full 500 ms.
Large messages
A record that is too large is rejected at the first limit it meets. The limits are checked in different places and at different sizes (before or after compression), so raising one is rarely enough.
| Where | Setting | Default | What it limits |
|---|---|---|---|
| Producer | max.request.size | 1048576 | Size of a request, and in effect the uncompressed size of a record batch. Exceeding it fails the send in the client |
| Producer | batch.size | 16384 | Not a limit on record size: a record larger than this is sent in a batch of its own |
| Broker | message.max.bytes | 1048588 | Largest record batch the broker accepts, after compression |
| Topic | max.message.bytes | 1048588 | The same limit for one topic; overrides the broker value |
| Broker | replica.fetch.max.bytes | 1048576 | Bytes per partition a follower tries to fetch. Not absolute: a larger first batch is still returned |
| Consumer | max.partition.fetch.bytes | 1048576 | Bytes per partition per fetch. Not absolute, for the same reason |
| Consumer | fetch.max.bytes | 52428800 | Bytes per fetch response. Not absolute |
max.request.size on the producer and max.message.bytes on the topic. The fetch limits let a single large batch through, but memory use follows.Before raising limits, ask whether the payload belongs in the log at all. A large record occupies broker memory and replication bandwidth, slows every consumer of that partition while it is fetched, and gets no benefit from batching. The usual alternative is to store the payload in object storage and publish an event that carries its location and checksum. Consumers that need the payload fetch it; the others never pay for it.
Architecture
Drawing diagram…
Flowchart of the tuning procedure: set a goal, take a baseline, calculate the per-partition byte rate, decide whether linger or batch size is the limit, change that one setting, measure under the same load, keep or revert, and repeat with compression until the goal is met.
flowchart TD
GOAL["Goal as numbers and fixed guarantees"] --> BASE["Baseline: explicit settings and metrics under repeatable load"]
BASE --> CALC["Per partition byte rate and time to fill a batch"]
CALC --> WHICH{"Fill time longer than linger.ms"}
WHICH -->|"yes, linger is the limit"| LINGER["Raise linger.ms within the latency budget"]
WHICH -->|"no, batches are full"| BATCH["Raise batch.size"]
LINGER --> MEASURE["Same load, same duration, compare metrics"]
BATCH --> MEASURE
COMP["Try compression once batches hold many records"] --> MEASURE
MEASURE --> BETTER{"Target metric improved"}
BETTER -->|"no"| REVERT["Revert the change"]
BETTER -->|"yes"| KEEP["Keep it and write down why"]
REVERT --> DONE{"Goal met"}
KEEP --> DONE
DONE -->|"no"| COMP
DONE -->|"yes"| STOP["Stop tuning"]- The calculation comes before any change. It tells you which setting can have an effect, so the first experiment is not wasted.
- Every path leads through the same measurement box: same load, same duration, same metrics. A change that is not measured is not finished.
- Reverting is a normal outcome. A setting that did not help should not stay in the configuration.
- The loop has an exit. Once the goal is met, further changes add risk without a reason.
Message flow
Drawing diagram…
Sequence diagram: the engineer runs a performance test with linger.ms 5 and reads the batch size metric, runs it again with only linger.ms changed to 50, compares the metrics, and then starts a third run that adds only compression.
sequenceDiagram participant E as Engineer participant T as Perf test producer participant B as Broker participant M as Producer metrics E->>T: run A with linger.ms 5 at fixed rate T->>B: many small produce requests B-->>T: acks T-->>M: batch-size-avg 1.2 KB E->>M: read baseline E->>T: run B with linger.ms 50, nothing else changed T->>B: fewer larger produce requests B-->>T: acks T-->>M: batch-size-avg 12 KB E->>M: compare batch size, queue time, p99 Note over E: keep the change if p99 is within budget E->>T: run C adds compression only
- Run A is the baseline. Its settings are written out, including the ones left at their defaults.
- Run B changes exactly one setting. The rate, record size, topic and duration are identical, so any difference in the metrics is due to that setting.
- The comparison uses several metrics together: batch size shows whether the change did what was predicted, queue time and p99 show what it cost.
- Run C starts from B, not from A. Each accepted change becomes the new baseline.
In a real system
A typical shop runs both kinds of stream. orders.events carries business events that a customer waits on. web.clicks carries page views keyed by session id to a 12-partition topic and feeds dashboards and recommendations a minute or more later. They get different producer configurations, in different KafkaTemplate beans or different services.
| Setting | orders.events | web.clicks | Reason |
|---|---|---|---|
acks | all | all | A guarantee, not a tuning choice. Lowering it for clicks is a business decision about losing data, to be made openly |
linger.ms | 5 | 50 | Orders: one record per batch at any linger, so do not add delay. Clicks: 4 records per batch becomes 42 |
batch.size | 16384 (default) | 32768 | Clicks: leaves room for traffic peaks above the 12.5 KB reached at linger 50 |
compression.type | none or lz4 | zstd or lz4 | Clicks: 42 similar JSON records per batch compress well. Orders: one record per batch gains little |
fetch.min.bytes (consumers) | 1 | 65536 | Clicks: about 8 fetches per second per partition instead of many small ones |
Worked example: how long does the buffer last?
| Quantity | Arithmetic | Result |
|---|---|---|
| Click producer instance, uncompressed | 10,000 records/s × 300 B | 3 MB/s |
buffer.memory default | 33,554,432 bytes | 33.5 MB |
| Time until the buffer is full if nothing can be sent | 33.5 / 3 | about 11 seconds |
| What happens next | send() blocks for up to max.block.ms (60,000), then fails with an exception | The calling thread is stuck for up to a minute |
| With a 4× larger buffer | 134 / 3 | about 45 seconds, then the same |
For clicks sent from a web request thread, blocking for a minute is worse than dropping the event, so that producer sets a short max.block.ms and counts failures. For orders the event is written through an outbox, so the request thread never calls send() at all (see the transactional outbox).
Java and Spring Boot
The first file is an A/B test with the tool shipped with Kafka, changing one setting between runs. The other two are the resulting Spring Boot configurations for the clickstream producer and its consumer, with every tuned value explicit.
# Sample payloads: one real click event per line, so compression is realistic.PAYLOADS=sample-clicks.jsonlCOMMON="--bootstrap-server localhost:9092 --topic perf-web-clicks --num-records 3000000 --payload-file $PAYLOADS --throughput 10000 --print-metrics"# Run A: baseline, every relevant setting written out.bin/kafka-producer-perf-test.sh $COMMON \ --command-property acks=all linger.ms=5 batch.size=16384 compression.type=none# Run B: only linger.ms differs from A.bin/kafka-producer-perf-test.sh $COMMON \ --command-property acks=all linger.ms=50 batch.size=16384 compression.type=none# Run C: only compression differs from B.bin/kafka-producer-perf-test.sh $COMMON \ --command-property acks=all linger.ms=50 batch.size=16384 compression.type=zstd# Run D: only batch.size differs from C.bin/kafka-producer-perf-test.sh $COMMON \ --command-property acks=all linger.ms=50 batch.size=32768 compression.type=zstdWhy it is written this way
--throughput 10000holds each run at the rate of one producer instance, so latency is measured at the real load. 3,000,000 records at that rate is a 5 minute run.--print-metricsprints the producer metrics at the end: comparebatch-size-avg,records-per-request-avg,record-queue-time-avg,compression-rate-avgandrequest-latency-avgbetween runs, together with the percentile line.- Each run differs from the previous one in a single property. If run D shows the same
batch-size-avgas run C, batches were not full and the largerbatch.sizeshould not be kept. - The test topic must match the real one in partitions, replication factor and
min.insync.replicas, and should be keyed like the real stream. The tool's--key-distributionand--record-key-rangeoptions (4.4) generate keys; without keys the sticky partitioner fills batches far better than a keyed stream would. - On releases before 4.x, pass properties with
--producer-props(includingbootstrap.servers). Check--helpon your version.
spring: kafka: bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS} producer: acks: all batch-size: 32KB compression-type: zstd buffer-memory: 32MB key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: enable.idempotence: true linger.ms: 50 max.block.ms: 2000 delivery.timeout.ms: 60000Why it is written this way
linger.mshas no dedicated Spring Boot property in 3.5, so it goes underproperties. It is set explicitly because the managed client (3.9) defaults to 0 and a 4.x client defaults to 5.batch-sizeandbuffer-memorytake a size with a unit. They map tobatch.sizeandbuffer.memory.max.block.ms: 2000bounds how long a web request thread can be held when the buffer is full or metadata is unavailable. The default is 60 seconds.delivery.timeout.msmust be at leastlinger.msplusrequest.timeout.ms(default 30000), or the producer refuses to start. 60 seconds here, instead of the default 2 minutes, means a click that cannot be delivered in that time is reported as failed and counted.acksandenable.idempotenceare written out so that nobody mistakes them for defaults that can be tuned away.
spring: kafka: bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS} consumer: group-id: click-aggregator fetch-min-size: 64KB fetch-max-wait: 200ms max-poll-records: 2000 properties: max.partition.fetch.bytes: 1048576 listener: type: batch concurrency: 4Why it is written this way
fetch-min-sizemaps tofetch.min.bytes: the broker answers once 64 KB is available. At 500 KB/s per partition that takes about 130 ms.fetch-max-wait: 200mscaps the wait at quiet times, instead of the default 500 ms.max-poll-records: 2000with a batch listener lets the aggregator write many clicks to its store in one statement. It changes whatpoll()returns, not how the consumer fetches.- The compressed batches are decompressed by this consumer, so compression chosen on the producer also costs CPU here.
Configuration
The settings this procedure touches, with defaults from the Kafka 4.4 references. Where the 3.9 client that Spring Boot 3.5 manages differs, it is noted.
linger.msDefault 5 (0 in clients before 4.0)
- What it does
- Longest time a record waits for its batch to fill before the batch is sent.
- When to change it
- Raise it when the per-partition byte rate fills less than a batch in the current linger and the stream tolerates the delay. Pointless when one record arrives per linger period.
batch.sizeDefault 16384
- What it does
- Upper bound in bytes of a batch for one partition. A buffer of this size is allocated for each batch.
- When to change it
- Raise it only when
batch-size-avgis close to it. Multiply by the number of partitions written to estimate memory.
compression.type (producer)Default none
- What it does
- Codec applied to whole batches by the producer: none, gzip, snappy, lz4 or zstd.
- When to change it
- Enable it when batches hold many similar records. Judge by
compression-rate-avgand CPU on real payloads. Keep the topic'scompression.typeatproducerso brokers do not recompress.
buffer.memoryDefault 33554432
- What it does
- Total memory for records waiting to be sent. When it is full,
send()blocks for up tomax.block.ms. - When to change it
- Raise it to absorb short bursts shown by
bufferpool-wait-ratio. If the buffer fills in steady state, the broker or the network is the limit and more memory only delays the error.
max.block.msDefault 60000
- What it does
- How long
send()may block waiting for metadata or buffer space before it fails. - When to change it
- Lower it where the calling thread serves a user, and handle the failure.
acksDefault all
- What it does
- Acknowledgements required before a request is complete. Idempotence requires
all. - When to change it
- Not a tuning setting. Changing it changes what an acknowledgement means.
max.in.flight.requests.per.connectionDefault 5
- What it does
- Unacknowledged requests the client sends on one connection before waiting. With idempotence, ordering is preserved for any value up to 5.
- When to change it
- Leave at 5 with idempotence on. Setting 1 is only needed for ordering when idempotence is off, and it reduces throughput.
max.request.sizeDefault 1048576
- What it does
- Largest request the producer sends, which also caps the uncompressed size of a record batch.
- When to change it
- Raise it together with the topic's
max.message.bytesif large records are truly needed. Prefer sending a reference to the payload.
max.message.bytes (topic)Default 1048588
- What it does
- Largest record batch the topic accepts, after compression. The broker-wide default is
message.max.bytes. - When to change it
- Raise per topic, not cluster-wide, so that one use case does not change limits for every topic.
fetch.min.bytesDefault 1
- What it does
- Minimum data the broker returns for a fetch; it waits up to
fetch.max.wait.msfor it. - When to change it
- Raise for high-volume streams where latency of tens to hundreds of milliseconds is acceptable.
fetch.max.wait.msDefault 500
- What it does
- Longest the broker holds a fetch that has not reached
fetch.min.bytes. - When to change it
- Set it to the added latency you can accept at quiet times.
max.partition.fetch.bytesDefault 1048576
- What it does
- Maximum data per partition per fetch; a larger first batch is still returned.
- When to change it
- Rarely needs changing. Lower it if a consumer with many partitions uses too much memory after a long pause.
max.poll.recordsDefault 500
- What it does
- Maximum records returned by one
poll(). Does not affect fetching. - When to change it
- Raise for batch listeners that write in bulk; lower when each record is slow, to keep the poll interval short.
Hands-on exercise
Find the limit, then move it
Use the producer settings simulator to predict and confirm which of linger.ms and batch.size limits a batch, changing one setting at a time.
Steps · 0 of 7 done
linger.ms against batch.size
Simulation · not a real clusterOutcomes are derived from the documented producer behaviour, not measured. Change one control, then read the batch picture and the throughput panel.
What you should see
A 16 KB batch needs 16 records, which takes 30 ms at one record every 2 ms. Below that, linger is the limit: 1 record at linger 0, 3 at 5 ms, 11 at 20 ms. At 40 and at 100 the batch is the same 16 records sent after 30 ms, because batch size is now the limit. With 64 KB at linger 100 the batch holds 51 records and is sent by the timer; at linger 200 it fills at 64 records after 126 ms. Back at linger 5, the batch is 3 records whatever the batch size.
What can go wrong
Raising batch.size on a stream that never fills a batch
- Symptom
batch.sizewas raised from 16 KB to 256 KB. Throughput andbatch-size-avgdid not change; producer memory use did.- Cause
- Batches were being sent by the linger timer long before reaching 16 KB, so the upper bound was never the limit.
- Fix
- Compare
batch-size-avgwithbatch.sizefirst. If the average is far below it, changelinger.msor accept that the stream is too thin to batch.
Many settings changed at once
- Symptom
- After a tuning session throughput is up, p99 latency is also up, CPU is higher, and nobody can say which change caused which effect.
- Cause
- Linger, batch size, compression and buffer size were changed together, so their effects cannot be separated.
- Fix
- One change per run under identical load. Keep a short log: setting, before, after, kept or reverted.
A bigger buffer to stop send() from blocking
- Symptom
send()blocked and timed out during peaks. Afterbuffer.memorywas quadrupled the errors come later in the peak and the application uses more heap.- Cause
- The application produces faster than the brokers accept for a sustained period. A buffer only postpones the moment it is full.
- Fix
- Find why sends are slow (
request-latency-avg, broker request metrics, quotas viaproduce-throttle-time-avg). Decide what the application does when it cannot send: slow down, shed load or fail the request.
Compression that compresses nothing
- Symptom
- CPU on producers and consumers rose after enabling compression.
compression-rate-avgis around 0.9 and network use barely changed. - Cause
- Batches hold one or two records, or the payload is already compressed or encrypted, so there is little redundancy to remove.
- Fix
- Get batches to hold many records first, test codecs on real payloads, and turn compression off where the ratio stays near 1.
Raising one size limit for a large record
- Symptom
RecordTooLargeExceptioncontinues aftermax.request.sizewas raised, now reported by the broker instead of the client.- Cause
- The producer limit was raised and the topic's
max.message.byteswas not. Each stage has its own limit. - Fix
- Raise the producer and topic limits together and check consumer memory. Better, store the payload outside Kafka and send a reference.
Benchmark settings carried into production
- Symptom
- A configuration copied from a load test has
acks=1and idempotence off. After a broker failure, acknowledged records are missing and some are duplicated. - Cause
- Guarantee settings were relaxed to get a higher benchmark figure and never restored.
- Fix
- Benchmark with the guarantees production needs. Keep
acks=allandenable.idempotence=trueexplicit in every configuration.
How to troubleshoot
Throughput is lower than expected and it is unclear whether batching works
Check
Producer metrics `batch-size-avg` and `records-per-request-avg`
- How to read it
records-per-request-avgnear 1 means no batching: the stream is thin per partition or linger is 0.batch-size-avgnearbatch.sizemeans batches are full and a larger size may help.
Records wait much longer than linger.ms before being sent
Check
Producer metrics `record-queue-time-avg`, `request-latency-avg` and `produce-throttle-time-avg`
- How to read it
- Queue time far above linger with high request latency means the broker responds slowly and batches queue behind in-flight requests. A throttle time above 0 means a quota is slowing the client.
send() blocks or throws after about a minute
Check
Producer metrics `buffer-available-bytes`, `bufferpool-wait-ratio` and `waiting-threads`
- How to read it
- Available bytes at 0 with threads waiting means the buffer is full: the producer cannot send as fast as the application produces. Blocking can also come from missing metadata (unknown topic, unreachable cluster).
The producer or broker rejects a record as too large
Check
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name orders.events --describe
- How to read it
- Shows topic overrides such as
max.message.bytes. If none is listed the broker'smessage.max.bytesapplies. Compare with the producer'smax.request.sizeand the size of the serialised record.
A consumer of a busy topic makes thousands of small fetches
Check
Consumer metrics `fetch-rate`, `fetch-size-avg` and `fetch-latency-avg`
- How to read it
- A high fetch rate with a small average size means each fetch returns as soon as a little data exists. Raising
fetch.min.bytestrades a bounded wait for fewer, larger fetches.
Key takeaways
- Calculate the per-partition byte rate per producer instance first. It tells you whether
linger.msorbatch.sizeis the limit, or whether batching can help at all. - Change one setting per run, under the same load, and keep the change only if the target metric improved.
- Read
batch-size-avg,records-per-request-avg,record-queue-time-avg,compression-rate-avgandbufferpool-wait-ratiobefore and after. acks, idempotence andmin.insync.replicasare guarantees. Do not trade them for a benchmark figure.- A larger
buffer.memoryabsorbs bursts and cannot fix a broker that is slower than the application. - A large record must pass the producer, topic and fetch limits. Sending a reference to the payload is usually the better design.
- Write tuned values down explicitly: defaults differ between client versions,
linger.msamong them.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A producer sends 600 records per second of 500 bytes each to a topic with 30 partitions, keyed by device id, from a single instance. linger.ms is 10 and batch.size is 16384. The team wants fewer, larger requests and proposes raising batch.size to 131072.
What will that change do?
Scenario 2 of 3
A team enables zstd compression on an audit topic. Each record is a 2 KB payload that the application has already encrypted. After the change, compression-rate-avg is 0.98, producer CPU is up by a fifth, and the bytes written to the brokers are almost unchanged.
What should they do?
Scenario 3 of 3
A document service starts publishing generated PDFs of up to 8 MB to a topic. The producer fails with RecordTooLargeException. An engineer raises max.request.size to 10 MB; now the broker rejects the records. The topic has three consumer groups, two of which only need the document id and status.
Which approach fits this situation best?
References
References: Apache Kafka 4.4: Producer configs (linger.ms, batch.size, compression.type, buffer.memory, max.block.ms, max.request.size, max.in.flight.requests.per.connection) ↗ · Apache Kafka 4.4: Consumer configs (fetch.min.bytes, fetch.max.wait.ms, max.partition.fetch.bytes, fetch.max.bytes, max.poll.records) ↗ · Apache Kafka 4.4: Broker configs (message.max.bytes, replica.fetch.max.bytes, compression.type) ↗ · Apache Kafka 4.4: Topic configs (max.message.bytes, compression.type) ↗ · Apache Kafka 4.4: Monitoring (producer and consumer fetch metrics) ↗ · Apache Kafka 4.4: Design (efficiency, end-to-end batch compression) ↗ · Apache Kafka 3.9: Producer configs (linger.ms default 0) ↗ · Spring Boot 3.5: Common application properties (spring.kafka.producer.*, spring.kafka.consumer.*) ↗ · Apache Kafka source, 4.4 branch: ProducerPerformance (options of kafka-producer-perf-test.sh) ↗ · Report a problem