Path 3 · lesson 4 of 5
Batching, linger.ms, compression and buffer.memory
How the producer groups records into per-partition batches, what linger.ms and batch.size trade, what happens when the buffer fills, and where compression is applied and undone.
What you will learn
- Explain when a batch for a partition is closed and sent, and which of linger.ms and batch.size is the limit in a given workload
- Estimate the batch size a producer will actually reach from its per-partition record rate
- Predict what send() does when buffer.memory is full and choose max.block.ms for the calling thread
- Say where a compressed batch is compressed, validated, stored and decompressed, and what a fixed topic codec costs the broker
- Read the producer metrics that show whether batching and compression are working
Before you start
The concept
send() does not send. It serialises the record, picks a partition and appends the record to a batch held in the producer's memory, then returns a future. A background sender thread later takes batches out of that memory and writes them to brokers. Everything in this lesson is about that gap between send() returning and the bytes leaving the process.
A batch belongs to one partition. The producer keeps a queue of batches for every partition it is writing to. A Produce request to a broker then carries one batch for each partition that broker leads and that has data ready. So there are two levels of grouping: records into a batch (per partition), and batches into a request (per broker).
| Setting | Default in 4.x | What it bounds |
|---|---|---|
linger.ms | 5 | How long a batch that is not yet full may wait for more records |
batch.size | 16384 bytes | The size a batch may grow to before it is sent regardless of the wait |
buffer.memory | 33554432 bytes (32 MB) | Total memory for records waiting to be sent, across all partitions |
max.block.ms | 60000 (1 minute) | How long send() may block waiting for metadata or for buffer space |
compression.type | none | The codec applied to each whole batch on the producer |
A batch is sent when the first of these happens: it reaches batch.size, it has waited linger.ms, or the application calls flush() or closes the producer. A record larger than batch.size is not split; it travels in a batch of its own.
Why it exists
Sending each record in its own request is expensive in ways that have nothing to do with the record. Each request has headers, a network round trip, a request handler on the broker, an append to the log and a response. Each batch has its own header and checksum. For a 200-byte event the fixed costs are comparable to the payload. A producer that sends record by record spends most of its effort, and most of the broker's, on overhead.
Batching removes that overhead by sharing it. Forty records in one batch pay for one batch header, one checksum, one append and one acknowledgement. The price is that the first record in the batch waits for the others, which is why the wait is bounded by linger.ms.
Compression has the same motivation applied to bytes. Events of one type repeat the same field names and many of the same values. Compressing one small record on its own finds little to remove. Compressing a batch of similar records together finds a lot. That is why Kafka compresses whole batches and why batching and compression are tuned together: larger batches compress better.
The buffer exists because the application and the network move at different speeds. Without a bounded buffer a slow or unreachable broker would either block every caller immediately or let memory grow until the process dies. buffer.memory and max.block.ms turn that into a defined behaviour: buffer up to a limit, then make the caller wait a bounded time, then fail the send.
How it works inside
How a batch forms
- The calling thread serialises key and value and chooses the partition (key hash, or the sticky partition for records without a key).
- It looks at the last batch in that partition's queue. If the record fits, it is appended there.
- If there is no open batch or the record does not fit, the producer takes a buffer of
batch.sizebytes from the pool thatbuffer.memorylimits and starts a new batch. - The sender thread repeatedly asks which batches are ready: full, older than
linger.ms, or flushed. It groups the ready batches by the broker that leads each partition and sends one Produce request per broker. - When the response arrives, every record in each batch gets the same outcome and the futures complete.
Two consequences follow. First, under load batching happens even with linger.ms=0: while a request is in flight, new records pile up behind it and leave together. linger.ms matters at low and moderate rates, where it is the only thing that makes records wait for each other. Second, the wait can be longer than linger.ms: when the broker is slow and the connection already has max.in.flight.requests.per.connection requests outstanding, batches stay in memory and keep growing. The documentation calls this out as broker backpressure producing a higher effective linger.
For records without a key, batch.size has a second role. The default partitioner sticks to one partition and moves to another only after at least batch.size bytes have been produced to it. That concentrates a keyless stream into full batches for one partition at a time instead of thin batches for all of them.
What limits a batch in practice
| Per-partition rate | Record size | linger.ms | Typical batch | Limit that closes it |
|---|---|---|---|---|
| 80 records/s | 300 B | 5 | 1 record, about 300 B | linger.ms |
| 80 records/s | 300 B | 50 | 4 records, about 1.2 KB | linger.ms |
| 2,000 records/s | 300 B | 5 | 10 records, about 3 KB | linger.ms |
| 2,000 records/s | 300 B | 50 | 54 records, 16 KB | batch.size, after about 27 ms |
| 2,000 records/s | 300 B | 50 with batch.size 65536 | 100 records, about 30 KB | linger.ms |
Raising batch.size does nothing if batches close on time before reaching it. Raising linger.ms does nothing if batches fill before the time is up. Find out which limit you are hitting before changing either; the metric batch-size-avg tells you.
When buffer.memory fills
Records leave the buffer only when a broker accepts them or when they fail. If the application appends faster than the sender can deliver (a slow broker, a partition without a leader, a network problem, a throttling quota), the pool runs out. From then on send() blocks the calling thread until space is freed. If no space appears within max.block.ms the send fails with BufferExhaustedException, which is a subclass of TimeoutException. The same max.block.ms budget also covers waiting for topic metadata at the start of send().
Time spent waiting in the buffer also counts against delivery.timeout.ms (default 2 minutes), which is the upper bound from send() returning to success or failure being reported. A record can therefore expire while still in memory, without ever having been sent. delivery.timeout.ms must be at least linger.ms plus request.timeout.ms.
Compression, end to end
- The producer compresses the records of a batch as one unit with the codec in
compression.type. The batch header, which records the codec, stays uncompressed. - The broker decompresses the batch to validate it, for example to check that the record count matches the header. If the topic's
compression.typeisproducer(the default), it appends the batch exactly as received. - The batch stays compressed on disk and is replicated to followers in that form.
- The batch is sent to consumers still compressed. The consumer reads the codec from the batch header and decompresses. Consumers need no configuration for this.
So one compression on the producer saves network between producer and broker, disk on every replica, network for replication, and network to every consumer group that reads the data. The cost is CPU on the producer, CPU on the broker for validation, and CPU on each consumer.
Topic compression.type = producer (default)
- The broker keeps whatever codec the producer used, including none
- Broker CPU is spent on validation only
- Different producers may write different codecs to the same topic; consumers handle each batch by its header
- The storage format is decided by client configuration
Topic compression.type = a fixed codec
- The broker stores every batch with that codec (
gzip,snappy,lz4,zstd) oruncompressed - Batches that arrive with a different codec are recompressed on the broker, on the produce path
- Useful to enforce a storage format when you do not control the producers
- Free only when producers already use the same codec
Choosing a codec
Kafka supports gzip, snappy, lz4 and zstd. The Kafka documentation does not rank them, and the result depends on your payload, so treat the table as a starting point and measure with your own data.
| Codec | General character | Level setting (default) |
|---|---|---|
lz4 | Very fast to compress and decompress, moderate size reduction. A common first choice when CPU matters more than bytes | compression.lz4.level (9, range 1 to 17) |
snappy | Similar goals to lz4: speed first, moderate reduction | no level setting |
zstd | Stronger reduction than lz4 or snappy at moderate CPU cost, tunable over a wide range | compression.zstd.level (3, range -131072 to 22) |
gzip | Strong reduction, the highest CPU cost of the four in most workloads | compression.gzip.level (-1, or 1 to 9) |
Payloads that are already compressed or encrypted (images, encrypted fields, some binary formats) gain almost nothing and still pay the CPU. Text formats such as JSON with repeated field names gain the most, and gain more as batches get larger.
Size limits meet compression
Two limits apply at different points. On the producer, max.request.size (default 1048576) caps a request and is in effect a cap on the uncompressed size of a batch. On the broker, message.max.bytes (default 1048588), or the topic's max.message.bytes, caps the batch after compression. A large record can therefore be rejected by the client even though its compressed form would have been accepted by the broker.
Batching and compression, side by side
Simulation · not a real clusterMove linger.ms and batch.size one at a time and watch which of them actually changes the batches. Then switch compression on and repeat.
Architecture
Drawing diagram…
A flowchart: application threads call send, records are serialised and placed in per-partition queues inside memory bounded by buffer.memory. A sender thread takes ready batches and sends one Produce request per broker, each carrying the batches for the partitions that broker leads. Brokers append the compressed batches and consumers decompress them.
flowchart LR
App["Application threads call send"] --> Part["Serialise and choose partition"]
subgraph Buf["Producer memory, bounded by buffer.memory"]
Q0["clicks-0 queue: open batch up to batch.size"]
Q1["clicks-1 queue: open batch"]
Q2["clicks-2 queue: open batch"]
end
Part --> Q0
Part --> Q1
Part --> Q2
Q0 --> Snd["Sender thread takes batches that are full or older than linger.ms"]
Q1 --> Snd
Q2 --> Snd
Snd --> R1["Produce request to broker 1 with batches for clicks-0 and clicks-2"]
Snd --> R2["Produce request to broker 2 with the batch for clicks-1"]
R1 --> B1["Broker 1 validates and appends the compressed batches"]
R2 --> B2["Broker 2 validates and appends the compressed batch"]
B1 --> Con["Consumers fetch compressed batches and decompress"]
B2 --> Con- Each partition has its own queue of batches. A batch never mixes partitions, which is why the per-partition record rate decides how full batches get.
- The memory box is one pool for the whole producer. A single partition that cannot be delivered (no leader, slow broker) can fill it and block sends to every other partition.
- The sender thread groups ready batches by destination broker. One request holds at most one batch per partition, and its total size is capped by
max.request.size. - The broker and the consumers see batches, not individual records. Compression, checksums, acknowledgement and retry all work on the whole batch.
Message flow
Drawing diagram…
A sequence diagram. The application thread sends forty records for one partition and gets futures back immediately. After linger.ms the sender thread takes the batch and sends one Produce request. The leader decompresses to validate, appends the batch as received, a follower fetches the same compressed batch, the leader acknowledges, all forty futures complete, and later a consumer fetches the compressed batch and decompresses it.
sequenceDiagram participant A as App thread participant P as Producer buffer participant S as Sender thread participant L as Partition leader participant F as Follower participant C as Consumer A->>P: send record 1 for clicks-3 P-->>A: future returned at once A->>P: send records 2 to 40 over the next few ms Note over P: one open batch for clicks-3, compressed with lz4 S->>P: linger.ms has passed, take the batch S->>L: Produce request with one compressed batch of 40 records L->>L: decompress to validate, assign offsets, append as received F->>L: Fetch L-->>F: the same compressed batch L-->>S: acknowledged once the in-sync replicas have it S-->>A: all 40 futures complete C->>L: Fetch L-->>C: compressed batch C->>C: decompress and deserialise 40 records
- The application thread never waits for the network here. It waits only if the buffer is full or metadata is missing, and then for at most
max.block.ms. - The acknowledgement covers the whole batch. With
acks=allit is sent when the replicas that are in sync at that moment have the batch; batching changes how many records each acknowledgement covers, not what it guarantees. - A retry resends the whole batch. With idempotence enabled the broker recognises a batch it already appended by its producer id and sequence numbers, so a retried batch is not written twice.
- The consumer needs no codec setting. It reads the codec from each batch header, so a topic can hold batches written with different codecs.
In a real system
A typical clickstream pipeline: ten instances of a web backend publish page and click events to the topic web.clickstream with 24 partitions. The key is the session id, so one session's events stay in order on one partition. Events are JSON of about 300 bytes. Each instance produces about 2,000 events a second. An analytics consumer group and a fraud-scoring consumer group both read the topic. These numbers are an illustration, not a measurement.
Per instance that is about 83 events a second for each partition. With a 3.x client and linger.ms=0, or with the 4.x default of 5, nearly every batch contains one event. Compression has one 300-byte JSON document to work with at a time and achieves little, and the brokers handle a very large number of tiny batches.
| Setting | Value for this topic | Reasoning |
|---|---|---|
linger.ms | 50 | Nobody waits on a click event. 50 ms gives about 4 events per batch per partition, and more at peak |
batch.size | 65536 | Not the limit at this rate, but leaves room when traffic is several times higher |
compression.type | zstd or lz4 | Repetitive JSON; two consumer groups and three replicas all benefit from each byte saved |
max.block.ms | 200 | A page view must not hang because analytics is slow; a failed send is counted and dropped |
buffer.memory | default (32 MB) | About 600 KB/s per instance, so 32 MB already absorbs tens of seconds of broker trouble |
Compare the orders.placed topic in the example project: a few hundred records a second in total, where the HTTP response waits for the acknowledgement. There a long linger would add delay to every checkout for little gain, so the default 5 ms, or a value of that order, is appropriate. The right settings come from the workload, not from the cluster.
Java and Spring Boot
A plain Java producer for the clickstream topic with the batching settings made explicit, followed by the same settings for a Spring Boot service and the topic-level compression setting.
package com.example.clicks;import java.time.Duration;import java.util.Properties;import org.apache.kafka.clients.producer.BufferExhaustedException;import org.apache.kafka.clients.producer.KafkaProducer;import org.apache.kafka.clients.producer.ProducerConfig;import org.apache.kafka.clients.producer.ProducerRecord;import org.apache.kafka.common.serialization.StringSerializer;import org.slf4j.Logger;import org.slf4j.LoggerFactory;public final class ClickEventProducer implements AutoCloseable { private static final Logger log = LoggerFactory.getLogger(ClickEventProducer.class); private static final String TOPIC = "web.clickstream"; private final KafkaProducer<String, String> producer; public ClickEventProducer(String bootstrapServers) { Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); props.put(ProducerConfig.CLIENT_ID_CONFIG, "web-backend-clicks"); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); // Batching: wait up to 50 ms for a batch of up to 64 KB per partition. props.put(ProducerConfig.LINGER_MS_CONFIG, 50); props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024); // Compression is applied to each whole batch. props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd"); // Bounded memory, and a short bound on how long a caller may be held. props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 32L * 1024 * 1024); props.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, 200); // Reliability settings stay at their defaults: acks=all, enable.idempotence=true. this.producer = new KafkaProducer<>(props); } /** Publishes one click event. Never holds the caller for longer than max.block.ms. */ public void publish(String sessionId, String eventJson) { ProducerRecord<String, String> record = new ProducerRecord<>(TOPIC, sessionId, eventJson); producer.send(record, (metadata, exception) -> { if (exception instanceof BufferExhaustedException) { // The buffer stayed full for max.block.ms: the brokers are not keeping up. log.warn("Click event dropped, producer buffer full"); } else if (exception != null) { log.warn("Click event not delivered", exception); } }); } @Override public void close() { // close() sends what is still buffered before returning, up to the timeout. producer.close(Duration.ofSeconds(10)); }}Why it is written this way
- One
KafkaProduceris created and shared by all threads. Batching depends on many callers appending to the same per-partition queues; a producer per request would never batch anything. linger.ms=50is chosen because no user waits for a click event. For a record whose acknowledgement an HTTP response waits on, 50 ms would be added to every request at low traffic.max.block.ms=200protects the web request threads. The trade is explicit: when the cluster is unreachable for longer than the buffer can absorb, click events are dropped and counted instead of stalling page views.- The callback distinguishes
BufferExhaustedExceptionbecause it means something different from a delivery failure: nothing was sent, and the cause is sustained backpressure. - The callback runs on the producer's sender thread. It only logs; slow work there delays every other batch.
close(Duration)on shutdown matters more with a long linger: records still waiting in memory are lost if the process exits without closing the producer.
spring: kafka: bootstrap-servers: localhost:9092 producer: client-id: web-backend-clicks batch-size: 65536 buffer-memory: 33554432 compression-type: zstd properties: "[linger.ms]": 50 "[max.block.ms]": 200Why it is written this way
- Spring Boot has first-class properties for
batch-size,buffer-memoryandcompression-type. There is no dedicated property forlinger.msormax.block.msin Spring Boot 3.5, so they go underspring.kafka.producer.properties. - The bracket syntax keeps the dotted Kafka name as one map key in YAML; it is the form the Spring Boot reference uses.
- Set
linger.msexplicitly in a Spring Boot 3.x service. Its managedkafka-clientsis a 3.x version, where the default is 0, not the 5 of a 4.x client.
# Show the overrides set on the topic (no compression.type line means the default, "producer").bin/kafka-configs.sh --bootstrap-server localhost:9092 \ --entity-type topics --entity-name web.clickstream --describe# Keep whatever codec producers send (the default). No recompression on the broker.bin/kafka-configs.sh --bootstrap-server localhost:9092 \ --entity-type topics --entity-name web.clickstream \ --alter --add-config compression.type=producer# Or enforce one storage codec. Batches arriving with another codec are recompressed by the broker.bin/kafka-configs.sh --bootstrap-server localhost:9092 \ --entity-type topics --entity-name web.clickstream \ --alter --add-config compression.type=zstdWhy it is written this way
- Leave the topic at
producerunless you have a reason to enforce a codec. Then the CPU cost of compression is paid once, by the client that produced the data. - If you do fix the topic codec, configure producers with the same codec so the broker has nothing to convert.
Configuration
Defaults are from the Apache Kafka 4.4 configuration reference and are the same from 4.0 onwards. The first eight are producer settings; the last two are set on the topic or broker.
linger.msDefault 5
- What it does
- Upper bound on the artificial delay added so that more records can join a batch. A batch that reaches
batch.sizeis sent without waiting. The default was 0 before Kafka 4.0. - When to change it
- Raise it (20 to 100) for high-volume streams nobody waits on: logs, metrics, clickstream. Keep it small where a caller waits for the acknowledgement. Setting 0 is rarely a latency win under load.
batch.sizeDefault 16384
- What it does
- Upper bound in bytes of a batch for one partition. A buffer of this size is allocated for each batch. Records larger than this are sent unbatched. Also the amount of data after which the sticky partitioner moves keyless records to another partition.
- When to change it
- Raise it (64 KB to a few hundred KB) only when
batch-size-avgis already close to the limit. Remember the memory cost: one buffer of this size per partition with an open batch.
buffer.memoryDefault 33554432
- What it does
- Total bytes the producer may use for records waiting to be sent. Not a hard bound on the producer's memory: compression and in-flight requests use more.
- When to change it
- Size it as bytes per second times the seconds of broker trouble you want to ride out, within the heap you have. A bigger buffer delays blocking; it does not prevent it.
max.block.msDefault 60000 (1 minute)
- What it does
- How long
send()may block in total waiting for metadata and for buffer space. After that the send fails; for a full buffer the exception isBufferExhaustedException. - When to change it
- Lower it to what the calling thread can afford, often well under a second for request threads, and handle the failure deliberately.
compression.typeDefault none
- What it does
- Codec for every batch this producer creates:
none,gzip,snappy,lz4orzstd. Whole batches are compressed, so more batching gives a better ratio. - When to change it
- Turn it on for text-like payloads at any real volume. Start with
lz4orzstdand comparecompression-rate-avgand CPU with your own data.
compression.zstd.levelDefault 3
- What it does
- Compression level used when
compression.typeiszstd. Valid range -131072 to 22. Available from Kafka 3.8. - When to change it
- Raise it when bytes cost more than producer CPU, and measure. The gain per step shrinks as the level rises.
compression.gzip.level / compression.lz4.levelDefault -1 / 9
- What it does
- Levels for
gzip(1 to 9, or -1 for the library default) andlz4(1 to 17). There is no level setting forsnappy. - When to change it
- Leave at the defaults unless a measurement with your payload shows a level that is clearly better for your CPU budget.
max.request.sizeDefault 1048576
- What it does
- Maximum size of a request, which limits how many batches go into one request and is in effect a cap on the uncompressed size of a batch.
- When to change it
- Raise it only together with the topic's
max.message.byteswhen you truly need larger records. For large payloads, storing the object elsewhere and sending a reference is usually the better design.
compression.type (topic; broker default of the same name)Default producer
- What it does
- The final codec of the topic.
producerkeeps the codec the producer used. A codec name oruncompressedmakes the broker store everything in that form. - When to change it
- Keep
producer. Set a fixed codec only to enforce a storage format, and then align the producers to avoid recompression on the broker.
max.message.bytes (topic; broker default message.max.bytes)Default 1048588
- What it does
- The largest record batch the broker accepts for the topic, measured after compression when compression is enabled.
- When to change it
- Raise it per topic when needed, not cluster-wide, and check that consumers' fetch sizes and the replica fetch size still make sense for larger batches.
Hands-on exercise
Find the limit that shapes your batches
Use the producer configuration simulator to see which of linger.ms and batch.size closes batches at a given load, and how compression depends on the result.
Steps · 0 of 7 done
Producer configuration simulator: throughput
Simulation · not a real clusterChange one setting at a time. The simulation is deterministic and conceptual: it shows the direction and rough size of each effect, not a benchmark of your cluster.
What you should see
With a short linger, raising batch.size alone changes nothing because batches close on time. Raising linger.ms grows batches and cuts requests until batch.size becomes the limit, after which only a larger batch.size helps. Compression pays off far more with fuller batches. The reliability panel does not move at all: batching and compression change cost and latency, not the delivery guarantee.
What can go wrong
Raising batch.size when linger.ms is the limit
- Symptom
- batch.size was increased to 512 KB, memory use went up, and throughput and request counts did not change.
- Cause
- At the per-partition record rate of the workload, batches are closed by
linger.mslong before they reach even 16 KB. The larger limit is never reached, but each open batch now reserves a larger buffer. - Fix
- Check
batch-size-avg. If it is far belowbatch.size, raiselinger.ms(or accept the batch size you have). Raisebatch.sizeonly when the average is near the limit.
Request threads blocked by a full buffer
- Symptom
- During a broker problem, HTTP latency jumps to exactly 60 seconds and then requests fail with a
TimeoutExceptionfromsend(). Thread pools fill and the service stops answering, including endpoints that do not use Kafka. - Cause
- The buffer filled because batches could not be delivered. Every
send()then blocked for the defaultmax.block.msof one minute on a request thread. - Fix
- Set
max.block.msto what a request can afford and decide what a failed send means: reject the request, or record the event durably elsewhere, or drop and count it. A largerbuffer.memoryonly postpones the same moment.
Compression enabled, but batches of one record
- Symptom
compression.typeis set, CPU went up slightly, andcompression-rate-avgstays close to 1. Disk and network use barely moved.- Cause
- Each batch holds one small record, so the codec has almost no repetition to remove. This is common with a 3.x client at
linger.ms=0and keyed records spread over many partitions. - Fix
- Get more records into each batch first (
linger.ms), then judge the codec. Also confirm the payload is compressible at all; already compressed or encrypted data will not shrink.
Topic codec differs from the producers' codec
- Symptom
- Broker CPU is high on the produce path and produce latency rises with traffic, while producers report normal compression.
- Cause
- The topic's
compression.typenames a fixed codec and producers send another one (or none). The broker decompresses and recompresses every batch to store it in the topic's codec. - Fix
- Set the topic to
producer, or make producers use the topic's codec. Decide in one place who owns the choice.
Records lost at shutdown with a long linger
- Symptom
- After a deployment, the last few hundred milliseconds of events from each instance are missing downstream. No errors were logged.
- Cause
- The process exited while records were still waiting in the producer's memory. With fire-and-forget sends nothing in the application was waiting for them.
- Fix
- Close the producer on shutdown (Spring's producer factory does this when the application context closes normally) and give the process time to do so. Keep
linger.msin mind when setting termination grace periods.
A large record rejected despite compression
- Symptom
RecordTooLargeExceptionon the producer for a 1.5 MB JSON document that would compress to 200 KB.- Cause
max.request.sizeis checked on the producer against the uncompressed record. The broker's limit applies to the compressed batch, but the record never gets that far.- Fix
- Prefer keeping large payloads out of Kafka and sending a reference. If you must send them, raise
max.request.sizeand the topic'smax.message.bytestogether and review consumer fetch sizes.
How to troubleshoot
Throughput is lower than expected and brokers see a very high request rate
Check
Producer metrics `batch-size-avg`, `records-per-request-avg` and `record-queue-time-avg` (MBean `kafka.producer:type=producer-metrics,client-id=<id>`)
- How to read it
batch-size-avgfar belowbatch.sizewith one or two records per request means batches close onlinger.ms.record-queue-time-avgis the time batches spent in the buffer; nearlinger.msis normal, far above it means the sender cannot keep up.
send() blocks or fails with BufferExhaustedException
Check
Producer metrics `buffer-available-bytes`, `bufferpool-wait-ratio`, `waiting-threads` and `buffer-exhausted-rate`
- How to read it
- Available bytes near zero and a wait ratio above zero mean callers are being held for buffer space. Then look for the reason batches are not leaving:
request-latency-avg,produce-throttle-time-avgfor quota throttling,record-retry-rate, and partition leadership on the brokers.
Compression seems to have no effect
Check
Producer metric `compression-rate-avg`
- How to read it
- It is the average ratio of compressed to uncompressed batch size, so lower is better and 1.0 means no reduction. Compare it at two
linger.msvalues before blaming the codec.
Need to compare settings before changing production
Check
bin/kafka-producer-perf-test.sh --topic perf-test --num-records 1000000 --record-size 300 --throughput -1 --producer-props bootstrap.servers=localhost:9092 linger.ms=50 batch.size=65536 compression.type=zstd
- How to read it
- Run it twice with one setting changed and compare records per second and latency percentiles. The generated payload is random bytes, which compress poorly; for compression tests pass realistic records with
--payload-file. Recent 4.x releases add--bootstrap-serverand prefer--command-propertyover--producer-props.
Unsure whether the broker is recompressing
Check
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name web.clickstream --describe
- How to read it
- If
compression.typeis listed with a codec that differs from the producers' setting, every batch is converted on the broker. No override listed means the broker default applies, which isproducerunless the cluster changed it.
Key takeaways
- A batch belongs to one partition. Batch size in practice is the per-partition record rate times
linger.ms, capped bybatch.size. linger.msbounds the wait,batch.sizebounds the size, and only the one you are actually hitting is worth changing. The 4.x default linger is 5 ms; 3.x clients default to 0.- When
buffer.memoryis full,send()blocks the caller for up tomax.block.msand then fails. Choose that bound for the calling thread and handle the failure. - Compression is applied by the producer to whole batches. The broker decompresses only to validate, stores and replicates the batch compressed, and consumers decompress it.
- Leave the topic's
compression.typeatproducerunless you need to enforce a codec; a different fixed codec makes the broker recompress every batch. - Batching and compression change cost and latency. They do not change what
acks, idempotence andmin.insync.replicasguarantee.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A service publishes keyed 400-byte JSON events at 240 records a second to a topic with 12 partitions. It runs with linger.ms=5, batch.size=16384 and compression.type=zstd. The producer metrics show batch-size-avg around 450 bytes and compression-rate-avg at 0.93. Nobody waits for these events.
Which single change is most likely to cut request count and improve compression?
Scenario 2 of 3
A checkout service calls send() on its HTTP request threads. During a 3-minute network partition from the Kafka cluster, every request hangs for 60 seconds and then fails, the request thread pool is exhausted and health checks time out. The producer runs with default buffer.memory and max.block.ms.
What should the team change first?
Scenario 3 of 3
A platform team sets compression.type=gzip on a busy topic to save disk. The producers, owned by other teams, are configured with compression.type=lz4. After the change, broker CPU on the partition leaders rises sharply and produce latency p99 doubles, while producer-side metrics look unchanged.
What explains this, and what is the cheapest remedy?
References
References: Apache Kafka 4.4: Producer configs (linger.ms, batch.size, buffer.memory, max.block.ms, compression.*, max.request.size) ↗ · Apache Kafka 4.4: Design, End-to-end batch compression ↗ · Apache Kafka 4.4: KafkaProducer Javadoc (buffering, max.block.ms, callbacks) ↗ · Apache Kafka 4.4: Topic configs (compression.type, max.message.bytes) ↗ · Apache Kafka 4.4: Message format, record batch attributes ↗ · Apache Kafka 4.4: Monitoring, producer metrics ↗ · KIP-390: Support Compression Level ↗ · Report a problem