Skip to content

Path 3 · lesson 4 of 5

Intermediate22 minApache Kafka 4.x (defaults checked against the 4.4 configuration reference; linger.ms is 5 since 4.0 and was 0 in 3.x clients)

Batching, linger.ms, compression and buffer.memory

How the producer groups records into per-partition batches, what linger.ms and batch.size trade, what happens when the buffer fills, and where compression is applied and undone.

What you will learn

  • Explain when a batch for a partition is closed and sent, and which of linger.ms and batch.size is the limit in a given workload
  • Estimate the batch size a producer will actually reach from its per-partition record rate
  • Predict what send() does when buffer.memory is full and choose max.block.ms for the calling thread
  • Say where a compressed batch is compressed, validated, stored and decompressed, and what a fixed topic codec costs the broker
  • Read the producer metrics that show whether batching and compression are working

Before you start

The concept

send() does not send. It serialises the record, picks a partition and appends the record to a batch held in the producer's memory, then returns a future. A background sender thread later takes batches out of that memory and writes them to brokers. Everything in this lesson is about that gap between send() returning and the bytes leaving the process.

A batch belongs to one partition. The producer keeps a queue of batches for every partition it is writing to. A Produce request to a broker then carries one batch for each partition that broker leads and that has data ready. So there are two levels of grouping: records into a batch (per partition), and batches into a request (per broker).

SettingDefault in 4.xWhat it bounds
linger.ms5How long a batch that is not yet full may wait for more records
batch.size16384 bytesThe size a batch may grow to before it is sent regardless of the wait
buffer.memory33554432 bytes (32 MB)Total memory for records waiting to be sent, across all partitions
max.block.ms60000 (1 minute)How long send() may block waiting for metadata or for buffer space
compression.typenoneThe codec applied to each whole batch on the producer
The five settings this lesson is about, with the defaults from the Kafka 4.x producer configuration reference.

A batch is sent when the first of these happens: it reaches batch.size, it has waited linger.ms, or the application calls flush() or closes the producer. A record larger than batch.size is not split; it travels in a batch of its own.

Why it exists

Sending each record in its own request is expensive in ways that have nothing to do with the record. Each request has headers, a network round trip, a request handler on the broker, an append to the log and a response. Each batch has its own header and checksum. For a 200-byte event the fixed costs are comparable to the payload. A producer that sends record by record spends most of its effort, and most of the broker's, on overhead.

Batching removes that overhead by sharing it. Forty records in one batch pay for one batch header, one checksum, one append and one acknowledgement. The price is that the first record in the batch waits for the others, which is why the wait is bounded by linger.ms.

Compression has the same motivation applied to bytes. Events of one type repeat the same field names and many of the same values. Compressing one small record on its own finds little to remove. Compressing a batch of similar records together finds a lot. That is why Kafka compresses whole batches and why batching and compression are tuned together: larger batches compress better.

The buffer exists because the application and the network move at different speeds. Without a bounded buffer a slow or unreachable broker would either block every caller immediately or let memory grow until the process dies. buffer.memory and max.block.ms turn that into a defined behaviour: buffer up to a limit, then make the caller wait a bounded time, then fail the send.

How it works inside

How a batch forms

  1. The calling thread serialises key and value and chooses the partition (key hash, or the sticky partition for records without a key).
  2. It looks at the last batch in that partition's queue. If the record fits, it is appended there.
  3. If there is no open batch or the record does not fit, the producer takes a buffer of batch.size bytes from the pool that buffer.memory limits and starts a new batch.
  4. The sender thread repeatedly asks which batches are ready: full, older than linger.ms, or flushed. It groups the ready batches by the broker that leads each partition and sends one Produce request per broker.
  5. When the response arrives, every record in each batch gets the same outcome and the futures complete.

Two consequences follow. First, under load batching happens even with linger.ms=0: while a request is in flight, new records pile up behind it and leave together. linger.ms matters at low and moderate rates, where it is the only thing that makes records wait for each other. Second, the wait can be longer than linger.ms: when the broker is slow and the connection already has max.in.flight.requests.per.connection requests outstanding, batches stay in memory and keep growing. The documentation calls this out as broker backpressure producing a higher effective linger.

For records without a key, batch.size has a second role. The default partitioner sticks to one partition and moves to another only after at least batch.size bytes have been produced to it. That concentrates a keyless stream into full batches for one partition at a time instead of thin batches for all of them.

What limits a batch in practice

Per-partition rateRecord sizelinger.msTypical batchLimit that closes it
80 records/s300 B51 record, about 300 Blinger.ms
80 records/s300 B504 records, about 1.2 KBlinger.ms
2,000 records/s300 B510 records, about 3 KBlinger.ms
2,000 records/s300 B5054 records, 16 KBbatch.size, after about 27 ms
2,000 records/s300 B50 with batch.size 65536100 records, about 30 KBlinger.ms
Arithmetic, not measurement: records per batch is roughly rate times linger, capped by batch.size divided by record size. Sizes are before compression.

Raising batch.size does nothing if batches close on time before reaching it. Raising linger.ms does nothing if batches fill before the time is up. Find out which limit you are hitting before changing either; the metric batch-size-avg tells you.

When buffer.memory fills

Records leave the buffer only when a broker accepts them or when they fail. If the application appends faster than the sender can deliver (a slow broker, a partition without a leader, a network problem, a throttling quota), the pool runs out. From then on send() blocks the calling thread until space is freed. If no space appears within max.block.ms the send fails with BufferExhaustedException, which is a subclass of TimeoutException. The same max.block.ms budget also covers waiting for topic metadata at the start of send().

Time spent waiting in the buffer also counts against delivery.timeout.ms (default 2 minutes), which is the upper bound from send() returning to success or failure being reported. A record can therefore expire while still in memory, without ever having been sent. delivery.timeout.ms must be at least linger.ms plus request.timeout.ms.

Compression, end to end

  1. The producer compresses the records of a batch as one unit with the codec in compression.type. The batch header, which records the codec, stays uncompressed.
  2. The broker decompresses the batch to validate it, for example to check that the record count matches the header. If the topic's compression.type is producer (the default), it appends the batch exactly as received.
  3. The batch stays compressed on disk and is replicated to followers in that form.
  4. The batch is sent to consumers still compressed. The consumer reads the codec from the batch header and decompresses. Consumers need no configuration for this.

So one compression on the producer saves network between producer and broker, disk on every replica, network for replication, and network to every consumer group that reads the data. The cost is CPU on the producer, CPU on the broker for validation, and CPU on each consumer.

Topic compression.type = producer (default)

  • The broker keeps whatever codec the producer used, including none
  • Broker CPU is spent on validation only
  • Different producers may write different codecs to the same topic; consumers handle each batch by its header
  • The storage format is decided by client configuration

Topic compression.type = a fixed codec

  • The broker stores every batch with that codec (gzip, snappy, lz4, zstd) or uncompressed
  • Batches that arrive with a different codec are recompressed on the broker, on the produce path
  • Useful to enforce a storage format when you do not control the producers
  • Free only when producers already use the same codec

Choosing a codec

Kafka supports gzip, snappy, lz4 and zstd. The Kafka documentation does not rank them, and the result depends on your payload, so treat the table as a starting point and measure with your own data.

CodecGeneral characterLevel setting (default)
lz4Very fast to compress and decompress, moderate size reduction. A common first choice when CPU matters more than bytescompression.lz4.level (9, range 1 to 17)
snappySimilar goals to lz4: speed first, moderate reductionno level setting
zstdStronger reduction than lz4 or snappy at moderate CPU cost, tunable over a wide rangecompression.zstd.level (3, range -131072 to 22)
gzipStrong reduction, the highest CPU cost of the four in most workloadscompression.gzip.level (-1, or 1 to 9)
The level settings exist on producers from Kafka 3.8 (KIP-390) and also as broker and topic settings for a fixed topic codec.

Payloads that are already compressed or encrypted (images, encrypted fields, some binary formats) gain almost nothing and still pay the CPU. Text formats such as JSON with repeated field names gain the most, and gain more as batches get larger.

Size limits meet compression

Two limits apply at different points. On the producer, max.request.size (default 1048576) caps a request and is in effect a cap on the uncompressed size of a batch. On the broker, message.max.bytes (default 1048588), or the topic's max.message.bytes, caps the batch after compression. A large record can therefore be rejected by the client even though its compressed form would have been accepted by the broker.

Batching and compression, side by side

Simulation · not a real cluster

Move linger.ms and batch.size one at a time and watch which of them actually changes the batches. Then switch compression on and repeat.

Architecture

Drawing diagram…

A flowchart: application threads call send, records are serialised and placed in per-partition queues inside memory bounded by buffer.memory. A sender thread takes ready batches and sends one Produce request per broker, each carrying the batches for the partitions that broker leads. Brokers append the compressed batches and consumers decompress them.

flowchart LR
  App["Application threads call send"] --> Part["Serialise and choose partition"]
  subgraph Buf["Producer memory, bounded by buffer.memory"]
    Q0["clicks-0 queue: open batch up to batch.size"]
    Q1["clicks-1 queue: open batch"]
    Q2["clicks-2 queue: open batch"]
  end
  Part --> Q0
  Part --> Q1
  Part --> Q2
  Q0 --> Snd["Sender thread takes batches that are full or older than linger.ms"]
  Q1 --> Snd
  Q2 --> Snd
  Snd --> R1["Produce request to broker 1 with batches for clicks-0 and clicks-2"]
  Snd --> R2["Produce request to broker 2 with the batch for clicks-1"]
  R1 --> B1["Broker 1 validates and appends the compressed batches"]
  R2 --> B2["Broker 2 validates and appends the compressed batch"]
  B1 --> Con["Consumers fetch compressed batches and decompress"]
  B2 --> Con
Records are grouped twice: into batches per partition in memory, then into one request per broker.
  • Each partition has its own queue of batches. A batch never mixes partitions, which is why the per-partition record rate decides how full batches get.
  • The memory box is one pool for the whole producer. A single partition that cannot be delivered (no leader, slow broker) can fill it and block sends to every other partition.
  • The sender thread groups ready batches by destination broker. One request holds at most one batch per partition, and its total size is capped by max.request.size.
  • The broker and the consumers see batches, not individual records. Compression, checksums, acknowledgement and retry all work on the whole batch.

Message flow

Drawing diagram…

A sequence diagram. The application thread sends forty records for one partition and gets futures back immediately. After linger.ms the sender thread takes the batch and sends one Produce request. The leader decompresses to validate, appends the batch as received, a follower fetches the same compressed batch, the leader acknowledges, all forty futures complete, and later a consumer fetches the compressed batch and decompresses it.

sequenceDiagram
  participant A as App thread
  participant P as Producer buffer
  participant S as Sender thread
  participant L as Partition leader
  participant F as Follower
  participant C as Consumer
  A->>P: send record 1 for clicks-3
  P-->>A: future returned at once
  A->>P: send records 2 to 40 over the next few ms
  Note over P: one open batch for clicks-3, compressed with lz4
  S->>P: linger.ms has passed, take the batch
  S->>L: Produce request with one compressed batch of 40 records
  L->>L: decompress to validate, assign offsets, append as received
  F->>L: Fetch
  L-->>F: the same compressed batch
  L-->>S: acknowledged once the in-sync replicas have it
  S-->>A: all 40 futures complete
  C->>L: Fetch
  L-->>C: compressed batch
  C->>C: decompress and deserialise 40 records
Forty records share one batch, one request, one append and one acknowledgement; the batch stays compressed until a consumer reads it.
  1. The application thread never waits for the network here. It waits only if the buffer is full or metadata is missing, and then for at most max.block.ms.
  2. The acknowledgement covers the whole batch. With acks=all it is sent when the replicas that are in sync at that moment have the batch; batching changes how many records each acknowledgement covers, not what it guarantees.
  3. A retry resends the whole batch. With idempotence enabled the broker recognises a batch it already appended by its producer id and sequence numbers, so a retried batch is not written twice.
  4. The consumer needs no codec setting. It reads the codec from each batch header, so a topic can hold batches written with different codecs.

In a real system

A typical clickstream pipeline: ten instances of a web backend publish page and click events to the topic web.clickstream with 24 partitions. The key is the session id, so one session's events stay in order on one partition. Events are JSON of about 300 bytes. Each instance produces about 2,000 events a second. An analytics consumer group and a fraud-scoring consumer group both read the topic. These numbers are an illustration, not a measurement.

Per instance that is about 83 events a second for each partition. With a 3.x client and linger.ms=0, or with the 4.x default of 5, nearly every batch contains one event. Compression has one 300-byte JSON document to work with at a time and achieves little, and the brokers handle a very large number of tiny batches.

SettingValue for this topicReasoning
linger.ms50Nobody waits on a click event. 50 ms gives about 4 events per batch per partition, and more at peak
batch.size65536Not the limit at this rate, but leaves room when traffic is several times higher
compression.typezstd or lz4Repetitive JSON; two consumer groups and three replicas all benefit from each byte saved
max.block.ms200A page view must not hang because analytics is slow; a failed send is counted and dropped
buffer.memorydefault (32 MB)About 600 KB/s per instance, so 32 MB already absorbs tens of seconds of broker trouble

Compare the orders.placed topic in the example project: a few hundred records a second in total, where the HTTP response waits for the acknowledgement. There a long linger would add delay to every checkout for little gain, so the default 5 ms, or a value of that order, is appropriate. The right settings come from the workload, not from the cluster.

Java and Spring Boot

A plain Java producer for the clickstream topic with the batching settings made explicit, followed by the same settings for a Spring Boot service and the topic-level compression setting.

src/main/java/com/example/clicks/ClickEventProducer.javaJava
package com.example.clicks;import java.time.Duration;import java.util.Properties;import org.apache.kafka.clients.producer.BufferExhaustedException;import org.apache.kafka.clients.producer.KafkaProducer;import org.apache.kafka.clients.producer.ProducerConfig;import org.apache.kafka.clients.producer.ProducerRecord;import org.apache.kafka.common.serialization.StringSerializer;import org.slf4j.Logger;import org.slf4j.LoggerFactory;public final class ClickEventProducer implements AutoCloseable {    private static final Logger log = LoggerFactory.getLogger(ClickEventProducer.class);    private static final String TOPIC = "web.clickstream";    private final KafkaProducer<String, String> producer;    public ClickEventProducer(String bootstrapServers) {        Properties props = new Properties();        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);        props.put(ProducerConfig.CLIENT_ID_CONFIG, "web-backend-clicks");        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);        // Batching: wait up to 50 ms for a batch of up to 64 KB per partition.        props.put(ProducerConfig.LINGER_MS_CONFIG, 50);        props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024);        // Compression is applied to each whole batch.        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");        // Bounded memory, and a short bound on how long a caller may be held.        props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 32L * 1024 * 1024);        props.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, 200);        // Reliability settings stay at their defaults: acks=all, enable.idempotence=true.        this.producer = new KafkaProducer<>(props);    }    /** Publishes one click event. Never holds the caller for longer than max.block.ms. */    public void publish(String sessionId, String eventJson) {        ProducerRecord<String, String> record = new ProducerRecord<>(TOPIC, sessionId, eventJson);        producer.send(record, (metadata, exception) -> {            if (exception instanceof BufferExhaustedException) {                // The buffer stayed full for max.block.ms: the brokers are not keeping up.                log.warn("Click event dropped, producer buffer full");            } else if (exception != null) {                log.warn("Click event not delivered", exception);            }        });    }    @Override    public void close() {        // close() sends what is still buffered before returning, up to the timeout.        producer.close(Duration.ofSeconds(10));    }}

Why it is written this way

  • One KafkaProducer is created and shared by all threads. Batching depends on many callers appending to the same per-partition queues; a producer per request would never batch anything.
  • linger.ms=50 is chosen because no user waits for a click event. For a record whose acknowledgement an HTTP response waits on, 50 ms would be added to every request at low traffic.
  • max.block.ms=200 protects the web request threads. The trade is explicit: when the cluster is unreachable for longer than the buffer can absorb, click events are dropped and counted instead of stalling page views.
  • The callback distinguishes BufferExhaustedException because it means something different from a delivery failure: nothing was sent, and the cause is sustained backpressure.
  • The callback runs on the producer's sender thread. It only logs; slow work there delays every other batch.
  • close(Duration) on shutdown matters more with a long linger: records still waiting in memory are lost if the process exits without closing the producer.

Configuration

Defaults are from the Apache Kafka 4.4 configuration reference and are the same from 4.0 onwards. The first eight are producer settings; the last two are set on the topic or broker.

linger.msDefault 5
What it does
Upper bound on the artificial delay added so that more records can join a batch. A batch that reaches batch.size is sent without waiting. The default was 0 before Kafka 4.0.
When to change it
Raise it (20 to 100) for high-volume streams nobody waits on: logs, metrics, clickstream. Keep it small where a caller waits for the acknowledgement. Setting 0 is rarely a latency win under load.
batch.sizeDefault 16384
What it does
Upper bound in bytes of a batch for one partition. A buffer of this size is allocated for each batch. Records larger than this are sent unbatched. Also the amount of data after which the sticky partitioner moves keyless records to another partition.
When to change it
Raise it (64 KB to a few hundred KB) only when batch-size-avg is already close to the limit. Remember the memory cost: one buffer of this size per partition with an open batch.
buffer.memoryDefault 33554432
What it does
Total bytes the producer may use for records waiting to be sent. Not a hard bound on the producer's memory: compression and in-flight requests use more.
When to change it
Size it as bytes per second times the seconds of broker trouble you want to ride out, within the heap you have. A bigger buffer delays blocking; it does not prevent it.
max.block.msDefault 60000 (1 minute)
What it does
How long send() may block in total waiting for metadata and for buffer space. After that the send fails; for a full buffer the exception is BufferExhaustedException.
When to change it
Lower it to what the calling thread can afford, often well under a second for request threads, and handle the failure deliberately.
compression.typeDefault none
What it does
Codec for every batch this producer creates: none, gzip, snappy, lz4 or zstd. Whole batches are compressed, so more batching gives a better ratio.
When to change it
Turn it on for text-like payloads at any real volume. Start with lz4 or zstd and compare compression-rate-avg and CPU with your own data.
compression.zstd.levelDefault 3
What it does
Compression level used when compression.type is zstd. Valid range -131072 to 22. Available from Kafka 3.8.
When to change it
Raise it when bytes cost more than producer CPU, and measure. The gain per step shrinks as the level rises.
compression.gzip.level / compression.lz4.levelDefault -1 / 9
What it does
Levels for gzip (1 to 9, or -1 for the library default) and lz4 (1 to 17). There is no level setting for snappy.
When to change it
Leave at the defaults unless a measurement with your payload shows a level that is clearly better for your CPU budget.
max.request.sizeDefault 1048576
What it does
Maximum size of a request, which limits how many batches go into one request and is in effect a cap on the uncompressed size of a batch.
When to change it
Raise it only together with the topic's max.message.bytes when you truly need larger records. For large payloads, storing the object elsewhere and sending a reference is usually the better design.
compression.type (topic; broker default of the same name)Default producer
What it does
The final codec of the topic. producer keeps the codec the producer used. A codec name or uncompressed makes the broker store everything in that form.
When to change it
Keep producer. Set a fixed codec only to enforce a storage format, and then align the producers to avoid recompression on the broker.
max.message.bytes (topic; broker default message.max.bytes)Default 1048588
What it does
The largest record batch the broker accepts for the topic, measured after compression when compression is enabled.
When to change it
Raise it per topic when needed, not cluster-wide, and check that consumers' fetch sizes and the replica fetch size still make sense for larger batches.

Hands-on exercise

Find the limit that shapes your batches

Use the producer configuration simulator to see which of linger.ms and batch.size closes batches at a given load, and how compression depends on the result.

Steps · 0 of 7 done

Producer configuration simulator: throughput

Simulation · not a real cluster

Change one setting at a time. The simulation is deterministic and conceptual: it shows the direction and rough size of each effect, not a benchmark of your cluster.

What can go wrong

Raising batch.size when linger.ms is the limit
Symptom
batch.size was increased to 512 KB, memory use went up, and throughput and request counts did not change.
Cause
At the per-partition record rate of the workload, batches are closed by linger.ms long before they reach even 16 KB. The larger limit is never reached, but each open batch now reserves a larger buffer.
Fix
Check batch-size-avg. If it is far below batch.size, raise linger.ms (or accept the batch size you have). Raise batch.size only when the average is near the limit.
Request threads blocked by a full buffer
Symptom
During a broker problem, HTTP latency jumps to exactly 60 seconds and then requests fail with a TimeoutException from send(). Thread pools fill and the service stops answering, including endpoints that do not use Kafka.
Cause
The buffer filled because batches could not be delivered. Every send() then blocked for the default max.block.ms of one minute on a request thread.
Fix
Set max.block.ms to what a request can afford and decide what a failed send means: reject the request, or record the event durably elsewhere, or drop and count it. A larger buffer.memory only postpones the same moment.
Compression enabled, but batches of one record
Symptom
compression.type is set, CPU went up slightly, and compression-rate-avg stays close to 1. Disk and network use barely moved.
Cause
Each batch holds one small record, so the codec has almost no repetition to remove. This is common with a 3.x client at linger.ms=0 and keyed records spread over many partitions.
Fix
Get more records into each batch first (linger.ms), then judge the codec. Also confirm the payload is compressible at all; already compressed or encrypted data will not shrink.
Topic codec differs from the producers' codec
Symptom
Broker CPU is high on the produce path and produce latency rises with traffic, while producers report normal compression.
Cause
The topic's compression.type names a fixed codec and producers send another one (or none). The broker decompresses and recompresses every batch to store it in the topic's codec.
Fix
Set the topic to producer, or make producers use the topic's codec. Decide in one place who owns the choice.
Records lost at shutdown with a long linger
Symptom
After a deployment, the last few hundred milliseconds of events from each instance are missing downstream. No errors were logged.
Cause
The process exited while records were still waiting in the producer's memory. With fire-and-forget sends nothing in the application was waiting for them.
Fix
Close the producer on shutdown (Spring's producer factory does this when the application context closes normally) and give the process time to do so. Keep linger.ms in mind when setting termination grace periods.
A large record rejected despite compression
Symptom
RecordTooLargeException on the producer for a 1.5 MB JSON document that would compress to 200 KB.
Cause
max.request.size is checked on the producer against the uncompressed record. The broker's limit applies to the compressed batch, but the record never gets that far.
Fix
Prefer keeping large payloads out of Kafka and sending a reference. If you must send them, raise max.request.size and the topic's max.message.bytes together and review consumer fetch sizes.

How to troubleshoot

Throughput is lower than expected and brokers see a very high request rate

Check

Producer metrics `batch-size-avg`, `records-per-request-avg` and `record-queue-time-avg` (MBean `kafka.producer:type=producer-metrics,client-id=<id>`)
How to read it
batch-size-avg far below batch.size with one or two records per request means batches close on linger.ms. record-queue-time-avg is the time batches spent in the buffer; near linger.ms is normal, far above it means the sender cannot keep up.
send() blocks or fails with BufferExhaustedException

Check

Producer metrics `buffer-available-bytes`, `bufferpool-wait-ratio`, `waiting-threads` and `buffer-exhausted-rate`
How to read it
Available bytes near zero and a wait ratio above zero mean callers are being held for buffer space. Then look for the reason batches are not leaving: request-latency-avg, produce-throttle-time-avg for quota throttling, record-retry-rate, and partition leadership on the brokers.
Compression seems to have no effect

Check

Producer metric `compression-rate-avg`
How to read it
It is the average ratio of compressed to uncompressed batch size, so lower is better and 1.0 means no reduction. Compare it at two linger.ms values before blaming the codec.
Need to compare settings before changing production

Check

bin/kafka-producer-perf-test.sh --topic perf-test --num-records 1000000 --record-size 300 --throughput -1 --producer-props bootstrap.servers=localhost:9092 linger.ms=50 batch.size=65536 compression.type=zstd
How to read it
Run it twice with one setting changed and compare records per second and latency percentiles. The generated payload is random bytes, which compress poorly; for compression tests pass realistic records with --payload-file. Recent 4.x releases add --bootstrap-server and prefer --command-property over --producer-props.
Unsure whether the broker is recompressing

Check

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name web.clickstream --describe
How to read it
If compression.type is listed with a codec that differs from the producers' setting, every batch is converted on the broker. No override listed means the broker default applies, which is producer unless the cluster changed it.

Key takeaways

  • A batch belongs to one partition. Batch size in practice is the per-partition record rate times linger.ms, capped by batch.size.
  • linger.ms bounds the wait, batch.size bounds the size, and only the one you are actually hitting is worth changing. The 4.x default linger is 5 ms; 3.x clients default to 0.
  • When buffer.memory is full, send() blocks the caller for up to max.block.ms and then fails. Choose that bound for the calling thread and handle the failure.
  • Compression is applied by the producer to whole batches. The broker decompresses only to validate, stores and replicates the batch compressed, and consumers decompress it.
  • Leave the topic's compression.type at producer unless you need to enforce a codec; a different fixed codec makes the broker recompress every batch.
  • Batching and compression change cost and latency. They do not change what acks, idempotence and min.insync.replicas guarantee.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A service publishes keyed 400-byte JSON events at 240 records a second to a topic with 12 partitions. It runs with linger.ms=5, batch.size=16384 and compression.type=zstd. The producer metrics show batch-size-avg around 450 bytes and compression-rate-avg at 0.93. Nobody waits for these events.

Which single change is most likely to cut request count and improve compression?

Scenario 2 of 3

A checkout service calls send() on its HTTP request threads. During a 3-minute network partition from the Kafka cluster, every request hangs for 60 seconds and then fails, the request thread pool is exhausted and health checks time out. The producer runs with default buffer.memory and max.block.ms.

What should the team change first?

Scenario 3 of 3

A platform team sets compression.type=gzip on a busy topic to save disk. The producers, owned by other teams, are configured with compression.type=lz4. After the change, broker CPU on the partition leaders rises sharply and produce latency p99 doubles, while producer-side metrics look unchanged.

What explains this, and what is the cheapest remedy?

References

References: Apache Kafka 4.4: Producer configs (linger.ms, batch.size, buffer.memory, max.block.ms, compression.*, max.request.size) ↗ · Apache Kafka 4.4: Design, End-to-end batch compression ↗ · Apache Kafka 4.4: KafkaProducer Javadoc (buffering, max.block.ms, callbacks) ↗ · Apache Kafka 4.4: Topic configs (compression.type, max.message.bytes) ↗ · Apache Kafka 4.4: Message format, record batch attributes ↗ · Apache Kafka 4.4: Monitoring, producer metrics ↗ · KIP-390: Support Compression Level ↗ · Report a problem