Skip to content

Path 3 · lesson 1 of 5

Intermediate22 minApache Kafka 4.x (producer configs checked against the 4.0, 4.1 and 4.4 reference) · Java 21 · Spring Boot 3.5 with Spring for Apache Kafka 3.3, which manage kafka-clients 3.9

Inside the producer: from send() to the partition log

What happens between KafkaTemplate.send() and a record landing in a partition: serializers, partition choice, the buffer of batches, the sender thread and the result.

What you will learn

  • Trace one record from KafkaTemplate.send() through serialisation, partition choice, the accumulator and the sender thread to the partition leader
  • Say which parts of a send run on your thread, which run on the producer's I/O thread, and where send() can block
  • Build a ProducerRecord with a key, headers and a timestamp, and publish it as JSON from Spring Boot
  • Handle the result of a send (success metadata or failure) instead of ignoring the returned future
  • Read the producer's buffer and error metrics to tell a full buffer from a slow broker

Before you start

The concept

A Kafka producer is a library inside your application, not a remote service you call. send() does not perform a network round trip. It turns your object into bytes, decides which partition the record belongs to, puts it into an in-memory batch for that partition, and returns. A background thread owned by the producer later ships whole batches to the brokers that lead those partitions and reports the outcome through a callback and a future.

The unit you hand over is a ProducerRecord. Only the topic and the value are mandatory in practice; everything else has a default.

FieldWhat it isIf you leave it out
topicName of the topic to append toRequired
keyBytes used to choose the partition, and the identity used by log compactionNo key: the producer spreads records over partitions (see Keys and partitioning)
valueThe payloadA null value is legal and means a tombstone on a compacted topic
headersName and byte-array pairs carried next to the payload (event type, trace id, event id)No headers
timestampEvent time in epoch millisecondsThe producer stamps the record with its current time
partitionAn explicit partition numberThe producer chooses the partition; this is the normal case
The timestamp you set is kept only when the topic uses CreateTime (the default). With LogAppendTime the broker overwrites it.

In a Spring Boot application you rarely touch KafkaProducer directly. KafkaTemplate wraps it: it obtains a producer from a ProducerFactory, calls send, and gives you a CompletableFuture<SendResult<K, V>>. Nothing about the mechanics below changes; the template is a thin layer over the same client.

Why it exists

The obvious design is one request per record: serialise, send, wait for the response, return. It is easy to reason about and it is slow. Each record pays a network round trip, every request carries its own framing overhead, the broker does many tiny writes, and your business thread is blocked on a socket for the whole time.

Kafka's log is built for sequential appends of many records at once, and a record batch is also the unit that is compressed, replicated and written to disk. So the producer is designed around batches. Your threads only fill buffers; one I/O thread turns the buffers into a small number of large requests. Records that arrive close together travel together without the application doing anything.

Blocking send per record

  • One request and one response per record
  • The caller's thread waits on the network
  • Throughput is limited by round-trip time
  • Failure is an exception at the call site

Buffered, asynchronous producer

  • One request carries batches for many partitions
  • The caller's thread only serialises and appends to memory
  • Throughput is limited by bytes per second, not by round trips
  • Failure arrives later, through the future or callback

The price of this design is the last line of the comparison: the result is asynchronous, so the application has to decide what a failed send means for the business operation that caused it. That decision is the part teams most often skip.

How it works inside

On your thread: everything up to the buffer

  1. Metadata. The producer needs to know the topic's partitions and which broker leads each one. The first send to a topic waits for that metadata and can block for up to max.block.ms (60 s by default). Afterwards the metadata is cached and refreshed every metadata.max.age.ms (5 minutes) or as soon as a broker answers with an error that says the cached view is stale, for example after a leader change.
  2. Serialisation. The configured key.serializer and value.serializer turn the key and value into byte arrays. Kafka itself only ever sees bytes. A serialiser that cannot handle the object throws SerializationException straight out of send(), on your thread.
  3. Partition choice. An explicit partition wins. Otherwise a key is hashed to a partition, and a record without a key goes to the partition the producer is currently filling. The details are in the next lesson.
  4. Size check. A record larger than max.request.size (1 MiB by default) is refused before it is buffered. The plain client reports this as a RecordTooLargeException on the returned future; KafkaTemplate rethrows it from send() (see below).
  5. Append to the accumulator. The record accumulator keeps a queue of batches per topic-partition. The record is appended to the open batch for its partition, or a new batch of batch.size bytes (16,384 by default) is allocated from a pool limited by buffer.memory (32 MiB by default). If the pool is empty because records are produced faster than they are sent, send() blocks until space is freed, again for at most max.block.ms.

On the sender thread: everything that touches the network

Each KafkaProducer owns one I/O thread (it shows up in thread dumps as kafka-producer-network-thread | <client.id>). It loops: find batches that are ready, group them by the broker that leads their partition, send one produce request per broker, and process responses. A batch is ready when it is full or when it has waited linger.ms. One request can carry a batch for each of several partitions on the same broker.

When the response arrives, the sender completes every record in the batch: the future gets a RecordMetadata (topic, partition, offset, timestamp) or an exception, and callbacks run. For retriable errors the batch goes back into the accumulator and is sent again; that loop, and what it does to ordering and duplicates, is the subject of Acks, retries and idempotence.

What KafkaTemplate adds

  • DefaultKafkaProducerFactory creates one producer and shares it between all callers when you are not using transactions. This matches the client's own advice: the producer is thread safe and one shared instance is generally faster than several.
  • send(...) overloads build the ProducerRecord for you from topic, partition, timestamp, key and value; send(ProducerRecord) takes one you built, which is what you need for headers.
  • The result is a CompletableFuture<SendResult<K, V>>. SendResult holds the ProducerRecord and the RecordMetadata. On failure the future completes with a KafkaProducerException whose cause is the Kafka exception.
  • Failures that are known before the record is buffered (no metadata within max.block.ms, buffer exhausted, record too large) are thrown by KafkaTemplate.send() itself as org.springframework.kafka.KafkaException with the message Send failed. So a send can fail in two places: at the call, or later through the future.
  • A ProducerListener can observe every send. The default LoggingProducerListener logs failures and does nothing on success, which is why failed sends appear in the log even if your code ignores them.

Architecture

Drawing diagram…

Flowchart of a Kafka producer. The application calls KafkaTemplate, which passes a ProducerRecord to the KafkaProducer. Inside it the record passes serializers and partition choice into the record accumulator, which holds one open batch per partition. A sender thread, guided by cached cluster metadata, sends batches to the brokers that lead each partition, and responses complete the callback and future.

flowchart LR
  APP["Application thread"] --> KT["KafkaTemplate send"]
  KT --> PR["ProducerRecord"]
  subgraph PRODUCER["KafkaProducer in your JVM"]
    SER["Key and value serializers"]
    PART["Partition choice"]
    ACC["Record accumulator"]
    B0["Open batch for partition 0"]
    B1["Open batch for partition 1"]
    SND["Sender thread"]
    META["Cached cluster metadata"]
  end
  PR --> SER
  SER --> PART
  PART --> ACC
  ACC --> B0
  ACC --> B1
  B0 --> SND
  B1 --> SND
  META -.-> PART
  META -.-> SND
  SND --> BR1["Broker 1 leads partition 0"]
  SND --> BR2["Broker 2 leads partition 1"]
  BR1 -.-> RES["Callback and future"]
  BR2 -.-> RES
Your thread works left of the accumulator; the sender thread owns everything to the right of it.
  • The accumulator is the boundary between two threads. Application threads write into it; the sender thread reads from it. That is why send() is fast and why its result is asynchronous.
  • Batches are per partition, requests are per broker. The sender uses the metadata to find the leader of each partition and combines batches that go to the same broker into one request.
  • Metadata feeds two decisions: the partition count is needed to choose a partition, and the leader map is needed to route the batch. Stale metadata shows up as retriable errors followed by a refresh, not as records in the wrong partition.
  • There is exactly one sender thread and one set of buffers per producer instance. Creating many producers multiplies threads, memory and broker connections without making any of them faster.

Message flow

Drawing diagram…

Sequence diagram. The order service calls KafkaTemplate, which calls KafkaProducer. The producer waits for metadata on first use, serialises, chooses a partition and appends the record to a batch in the accumulator, then returns a future. Later the sender thread drains ready batches, sends a produce request to the partition leader, receives the response and completes the future with the partition and offset.

sequenceDiagram
  participant App as Order service thread
  participant KT as KafkaTemplate
  participant KP as KafkaProducer
  participant Acc as Record accumulator
  participant Snd as Sender thread
  participant B as Partition leader
  App->>KT: send record
  KT->>KP: send record with callback
  KP->>KP: wait for topic metadata on first use
  KP->>KP: serialize key and value
  KP->>KP: choose partition
  KP->>Acc: append to the batch for that partition
  KP-->>KT: Future
  KT-->>App: CompletableFuture of SendResult
  Note over Acc,Snd: batch becomes ready when full or after linger.ms
  Snd->>Acc: drain ready batches
  Snd->>B: Produce request with batches
  B-->>Snd: Produce response with base offset
  Snd->>KT: run callback
  KT-->>App: future completes with partition and offset
One send: the call returns after the append; the result arrives later from the sender thread.
  1. The first eight messages happen on the caller's thread and usually take microseconds. The exception is the very first send to a topic, which waits for metadata.
  2. Between the future being returned and the produce request there is a deliberate gap of up to linger.ms, so that more records can join the batch.
  3. The broker's response carries the base offset of the batch. The producer derives each record's offset from it and completes the futures in order; for one partition, callbacks are guaranteed to run in the order the records were sent.
  4. If the response is an error or never arrives, the last two messages are replaced by a retry or by the future completing with an exception.

In a real system

A typical order service publishes an OrderPlaced event when a customer checks out. In the example project this is the first hop: Order Service → Kafka → Payment Service.

DecisionChoiceReason
Topicorders.order-placed.v1, 6 partitions, replication factor 3One event type per topic keeps the JSON contract simple; six partitions allow six payment consumers
KeyorderId as a stringEvery event about one order lands in one partition and stays in order
ValueThe OrderPlaced record as JSONReadable in tools and by non-Java consumers; a schema registry comes later in the course
Headersevent-id, event-typeConsumers can deduplicate and route without parsing the payload
TimestampWhen the order was placedEvent time, so retention and time-based lookups follow the business time rather than the send time
ProducerOne shared producer per service instance, client.id=order-serviceOne set of buffers and connections; the client id appears in broker logs, quotas and metric names

At 200 orders per second and roughly 400 bytes per event, the producer buffers about 80 KB per second across six partitions. Batches fill slowly, so most are sent because linger.ms expired rather than because they were full. This is a low-volume producer; the settings that matter here are the ones about reliability, not throughput.

Java and Spring Boot

A Spring Boot 3 producer that publishes OrderPlaced as JSON, keyed by order id, and handles the outcome of every send. Spring Boot auto-configures the ProducerFactory and the KafkaTemplate from spring.kafka.*.

src/main/java/com/example/orders/events/OrderPlaced.javaJava
package com.example.orders.events;import java.math.BigDecimal;import java.time.Instant;/** * The event contract. Consumers depend on these field names, so treat * a rename or a type change as a breaking change to the topic. */public record OrderPlaced(        String eventId,        String orderId,        String customerId,        BigDecimal total,        String currency,        Instant placedAt) {}

Why it is written this way

  • A record gives an immutable event with no boilerplate, and Jackson serialises records without extra configuration.
  • eventId identifies this event, not the order. Consumers use it to recognise a redelivered event; the producer cannot do that for them.
  • placedAt is the business time. It is also used as the Kafka record timestamp in the publisher.

Configuration

The settings that shape the path from send() to the network. Defaults are from the Apache Kafka 4.x producer configuration reference; batching is covered in depth in its own lesson.

bootstrap.serversDefault none (required)
What it does
Host and port pairs used only for the first connection. The producer then discovers every broker from metadata.
When to change it
List two or three brokers so startup does not depend on one machine. It does not need to be the whole cluster.
key.serializer / value.serializerDefault none (required)
What it does
Classes that turn the key and the value into bytes.
When to change it
Pick the key serialiser once and keep it: the partition is computed from the serialised key bytes, so changing it moves keys to other partitions.
client.idDefault empty string
What it does
A logical name sent with every request. It appears in broker request logs, in quotas and in the producer's metric and thread names.
When to change it
Set it to the service name. Without it you cannot tell producers apart on the broker side.
buffer.memoryDefault 33554432 (32 MiB)
What it does
Total bytes the producer may use to buffer records that have not been sent yet.
When to change it
Raise it only if bursts are larger than the buffer and you have measured that. A buffer that is constantly full means the brokers cannot keep up, and more memory only delays the block.
max.block.msDefault 60000 (1 minute)
What it does
How long send() may block waiting for metadata or for buffer space before it fails.
When to change it
Lower it when send() is called on a request thread, so that a broker outage fails fast instead of holding every request for a minute.
batch.sizeDefault 16384
What it does
Upper bound, in bytes, of one batch for one partition. A full batch is sent immediately.
When to change it
Leave it for low volumes. Tune it together with linger.ms for throughput; see Batching and compression.
linger.msDefault 5 (0 in clients before 4.0)
What it does
How long a batch that is not full waits for more records before it is sent.
When to change it
A few milliseconds usually cost nothing and reduce request count. Set it explicitly if you run mixed client versions.
max.request.sizeDefault 1048576 (1 MiB)
What it does
Largest request the producer will send, which is also the limit for one serialised record.
When to change it
Raising it also requires raising the topic's max.message.bytes. Large payloads usually belong in object storage with a reference in the event.
metadata.max.age.msDefault 300000 (5 minutes)
What it does
Interval after which the producer refreshes metadata even without an error, to learn about new brokers and partitions.
When to change it
Rarely changed. It explains why a producer may take minutes to start using newly added partitions.
spring.kafka.producer.properties.spring.json.add.type.headersDefault true
What it does
Whether Spring's JsonSerializer adds headers with the Java type of the payload.
When to change it
Set to false when consumers are not sharing your classes, and tell consumers which type to expect in their own configuration.

Hands-on exercise

Watch the buffer hold records back

Use the producer simulation to see that records wait in a batch before they are sent, and how the size and time limits decide when a batch leaves.

Steps · 0 of 5 done

Producer buffer and batches

Simulation · not a real cluster

Change linger.ms and batch.size and watch when batches are sent. This is a deterministic simulation, not a real broker.

Take it further in the lab: Producer publishing messages

What can go wrong

Ignoring the send result
Symptom
The service reports success, but some events never appear on the topic. The only trace is an error from LoggingProducerListener that nobody alerts on.
Cause
kafkaTemplate.send(...) was called and the returned future dropped. A failure after send() returned has nowhere to go.
Fix
Attach whenComplete or wait on the future with a timeout, and make the failure change something: fail the request, mark the row as unpublished, or use an outbox so publishing is retried from the database.
Creating a producer per request
Symptom
Thread count, memory and broker connections grow with traffic; brokers show thousands of short-lived client connections.
Cause
Each KafkaProducer has its own buffer pool, sender thread and connections. Building one per request or per message pays that cost every time and never batches.
Fix
Share one producer. With Spring, inject the KafkaTemplate and let DefaultKafkaProducerFactory keep its single producer. The Break card A producer for every request shows where this leads.
Blocking work in the callback
Symptom
Throughput drops to a few dozen records per second and record-queue-time-avg climbs, while brokers are idle and request-latency-avg is normal.
Cause
The callback does a database write or a remote call. It runs on the single sender thread, so no other batch is sent until it returns.
Fix
Keep the callback to logging and counters, or continue on your own executor with whenCompleteAsync(action, executor).
Exiting without closing the producer
Symptom
The last few records before a shutdown or at the end of a batch job are missing.
Cause
They were still in the accumulator. Killing the process discards the buffer.
Fix
Let Spring shut down normally so the producer factory closes the producer, which sends what is buffered. In a short-lived job, call kafkaTemplate.flush() or wait for the futures before exiting.
Changing the serialiser on a live topic
Symptom
Consumers start failing to deserialise, or events for the same entity appear in a different partition than before.
Cause
Kafka stores bytes and does not know the format. A new value format breaks readers; a new key format changes the bytes that are hashed.
Fix
Treat key and value formats as the topic's contract. Introduce a new topic or a compatible schema change instead of swapping serialisers in place.

How to troubleshoot

send() hangs, then throws KafkaException: Send failed caused by TimeoutException: Topic orders.order-placed.v1 not present in metadata after 60000 ms.

Check

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders.order-placed.v1
How to read it
The producer could not get metadata within max.block.ms. If the command also fails, the bootstrap address, listener configuration or network is wrong. If it reports that the topic does not exist, create it (auto-creation may be disabled). If it works from your shell, compare the address and security settings the application uses.
send() blocks for seconds under load and sometimes fails with BufferExhaustedException

Check

Producer metrics `buffer-available-bytes`, `waiting-threads` and `bufferpool-wait-ratio` (MBean `kafka.producer:type=producer-metrics,client-id=order-service`)
How to read it
Available bytes near zero with waiting threads above zero means records are produced faster than the sender can ship them. Look at request-latency-avg next: if it is high, the brokers are slow and more buffer will not help.
SerializationException: Can't convert value of class ... to class ...StringSerializer specified in value.serializer

Check

The `ProducerConfig values` block the client logs at startup, lines `key.serializer` and `value.serializer`
How to read it
The configured serialiser does not match the type passed to send(). The generic types on KafkaTemplate are not checked against configuration at compile time; the configured classes decide.
Send fails with RecordTooLargeException: The message is N bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.

Check

kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name orders.order-placed.v1 --describe
How to read it
The record exceeds the producer's limit. Look at the topic's max.message.bytes too, since the broker enforces its own limit. Decide whether the payload should be that large before raising both.
Records are sent but arrive later than expected

Check

Producer metrics `record-queue-time-avg`, `batch-size-avg` and `request-latency-avg`
How to read it
Queue time is how long batches wait in the accumulator. Queue time close to linger.ms with small batches is the linger working as designed; queue time far above it means the sender is busy, throttled or blocked by a callback.

Key takeaways

  • send() serialises, picks a partition and appends to an in-memory batch. It returns before anything reaches a broker.
  • One sender thread per producer turns batches into requests, one request per broker, and completes futures and callbacks.
  • A batch leaves when it reaches batch.size or has waited linger.ms, whichever comes first.
  • send() can still block: for metadata on first use and for buffer space when the producer is outpacing the brokers, both bounded by max.block.ms.
  • Share one producer per application. KafkaTemplate with the default factory already does.
  • The future is the only place a late failure shows up. Decide what the business operation does when it fails.
  • Producer defaults come from the client library version in your application, not from the broker.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A checkout endpoint saves the order, calls kafkaTemplate.send(...) and returns 201 without looking at the result. During a three-minute broker outage the application log shows Expiring 12 record(s) for orders.order-placed.v1-3:120000 ms has passed since batch creation. All twelve customers received a 201 and no payment was ever started.

Which change addresses the actual defect?

Scenario 2 of 3

Under a traffic peak, request threads in a service stall for up to 60 seconds inside KafkaTemplate.send and then fail. Producer metrics show buffer-available-bytes near 0, waiting-threads at 40 and request-latency-avg at 900 ms, up from 15 ms. Heap usage is fine.

What is the most accurate reading of these numbers?

Scenario 3 of 3

A team adds a whenComplete callback that updates an outbox row with a JDBC call taking about 40 ms. Afterwards the producer sends roughly 25 records per second regardless of load, record-queue-time-avg rises to several seconds, and broker CPU is low.

What should they change first?

References

References: Apache Kafka documentation: Producer Configs ↗ · KafkaProducer Javadoc (buffer, I/O thread, send, callbacks) ↗ · ProducerRecord Javadoc ↗ · Apache Kafka documentation: Monitoring (producer metrics) ↗ · Spring for Apache Kafka 3.3 reference: Sending Messages ↗ · Spring for Apache Kafka 3.3 reference: Serialization, Deserialization, and Message Conversion ↗ · Spring Boot 3.5 reference: Apache Kafka Support ↗ · Report a problem