Skip to content

Path 3 · lesson 5 of 5

Intermediate30 minJava 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 · kafka-clients 3.9 (the version Spring Boot 3.5 manages) against Apache Kafka 4.x brokers

A working Spring Boot producer, and what its settings change

Build a small order service that publishes keyed events with KafkaTemplate and handles the send result, then change its settings and see what moves: throughput or reliability.

What you will learn

  • Build a Spring Boot service that publishes a keyed JSON event with KafkaTemplate and reports the outcome to its caller
  • Handle both ways a send can fail: an exception thrown by send() and a future that completes exceptionally
  • Set producer configuration through spring.kafka.producer.* and know which defaults come from Spring Boot and which from the Kafka client
  • Predict whether a configuration change affects throughput, latency, durability, duplicates or ordering
  • Recognise when publishing an event together with a database write needs an outbox instead of a direct send

Before you start

The concept

This lesson puts the producer path together as one small service. POST /orders accepts an order, publishes an OrderPlaced event to the topic orders.placed keyed by the order id, waits for the broker's acknowledgement and answers the caller according to what happened. It is the first step of the example project, reduced to the producer.

With spring-kafka on the classpath, Spring Boot creates the pieces for you from spring.kafka.* properties: a ProducerFactory that owns the Kafka producer, a KafkaTemplate that sends through it, and a KafkaAdmin that creates any NewTopic beans at startup. You inject KafkaTemplate and call send. By default the factory creates one producer shared by all threads, which is what the Kafka client recommends and what makes batching work.

DecisionChoice in this serviceWhy
Record keyorderIdAll events about one order go to one partition and stay in order relative to each other
Value formatJSON via JsonSerializer, no type headersReadable and simple; consumers in other services must not depend on this service's Java class name
Acknowledgementacks=all with idempotenceAn acknowledged order event survives a broker failure, and the producer's own retries do not create duplicates or reorder
OutcomeWait for the future with a timeoutThe HTTP response must not claim success for an event that was never stored
Time boundsmax.block.ms 5 s, delivery.timeout.ms 30 sA request thread must never be held for the one-minute and two-minute defaults
Batchinglinger.ms 10, lz4A small wait that a checkout can afford; set explicitly because the client default differs between versions
Every row is a choice you can defend or change. The rest of the lesson explains each one.

Why it exists

A producer that appears to work takes one line: kafkaTemplate.send("orders.placed", event). In a demo the record arrives. The problems are in what that line leaves undecided.

  • No key. Events for the same order are spread over partitions, so a consumer can see OrderCancelled before OrderPlaced.
  • No outcome. send returns a future immediately. If nobody looks at it, a failed send leaves only a log line from the template's default LoggingProducerListener. The HTTP caller was already told the order was accepted.
  • No time bound. With defaults, a send can block the calling thread for up to a minute waiting for metadata or buffer space, and a record can stay undecided for two minutes.
  • No stated configuration. The behaviour is whatever the client version's defaults are, and those have changed across versions.
  • No plan for the database. As soon as the order is also saved to a database, sending directly creates two writes that can disagree.

None of these is advanced. They are the difference between an example and a service you can run, and each has a small, concrete answer in the code below.

How it works inside

What happens inside one send

  1. KafkaTemplate.send(topic, key, value) builds a ProducerRecord and obtains the shared producer from the ProducerFactory.
  2. On the calling thread, the producer serialises the key (StringSerializer) and the value (JsonSerializer), chooses the partition from the hash of the key, and appends the record to that partition's open batch. If metadata is missing or the buffer is full, this is where the thread waits, for at most max.block.ms.
  3. send returns a CompletableFuture<SendResult<K, V>>. Nothing has reached a broker yet.
  4. The producer's sender thread sends the batch. With acks=all the partition leader answers once the replicas that are in sync at that moment have the batch.
  5. The producer's callback runs on the sender thread and completes the future: with a SendResult (the record and its RecordMetadata: partition and offset) or exceptionally. Functions attached with whenComplete run there too.

The ways a send fails

WhereExamplesHow it reaches youIs the record in Kafka?
Before bufferingThe value cannot be serialisedsend throws on the calling threadNo
Waiting to bufferTopic metadata unavailable, or buffer full for max.block.msThe send fails with a TimeoutException (BufferExhaustedException for a full buffer)No
Rejected by the brokerRecordTooLargeException, authorisation failureFuture completes exceptionally at once; these are not retriedNo
Retries run out of timeLeader unavailable or too few in-sync replicas for longer than delivery.timeout.msFuture completes exceptionally with a TimeoutExceptionNormally no, but a request whose response was lost may have been appended
Your own wait ends firstfuture.get(5, SECONDS) times outjava.util.concurrent.TimeoutException from getUnknown: the producer is still trying
With Spring's template, a failed future carries a KafkaProducerException; its getFailedProducerRecord() returns the record and its cause is the Kafka exception.

The last two rows are the ones that need thought. A timeout is not a clean failure. The honest response to the caller is “not confirmed, try again”, and the design must then tolerate the same order being submitted twice.

What to do when a send fails

Kind of eventReasonable reactionCost
The request is meaningless without it (this service)Fail the request and let the caller retry with the same idAvailability of the endpoint now depends on Kafka
Must never be lost, and belongs to a database changeDo not send directly: write it to an outbox table in the same database transactionAn outbox table and a relay or CDC connector to operate
Useful but not essential (metrics, clickstream)Count the failure and drop the eventSome data loss during incidents, by decision

What each setting changes

ChangeThroughput and latencyReliability
acks=all to acks=1Slightly lower produce latency: the leader no longer waits for followersAcknowledged records can be lost if the leader fails before followers copy them. Also conflicts with idempotence
acks=0Lowest latency; nothing is awaitedNo acknowledgement, no retries, the returned offset is always -1. Loss is silent
Idempotence off, retries on, up to 5 in flightNo meaningful gainRetries can write a batch twice and can put a later batch before an earlier one
linger.ms 0 to 10 to 100Fewer, fuller requests and better compression; each record may wait that much longer at low trafficUnchanged
compression.type none to lz4 or zstdFewer bytes on the network and on disk for some CPU; depends on batch fillUnchanged
delivery.timeout.ms lowerFailures are reported soonerLess time to ride out a leader election before the send fails
Batching and compression move cost and latency. acks, idempotence and retries move the guarantee. They are largely independent.

acks=all is a durability guarantee only together with the topic's replication factor and min.insync.replicas. With replication factor 3 and min.insync.replicas=2, an acknowledged record is on at least two brokers, and one broker can fail without loss or loss of write availability. With min.insync.replicas=1, acks=all can be satisfied by the leader alone.

Architecture

Drawing diagram…

A flowchart of the order service. An HTTP client calls OrderController, which calls OrderEventPublisher, which uses KafkaTemplate. Spring Boot creates the template, a DefaultKafkaProducerFactory with one shared KafkaProducer, and a KafkaAdmin. The producer serialises, batches per partition and its sender thread writes to the topic orders.placed, which payment and inventory consumer groups read. A NewTopic bean is created on the cluster by KafkaAdmin.

flowchart LR
  Client["HTTP client"] --> Ctl["OrderController"]
  Ctl --> Pub["OrderEventPublisher"]
  Pub --> Tpl["KafkaTemplate"]
  subgraph Boot["Created by Spring Boot from spring.kafka properties"]
    Tpl --> PF["DefaultKafkaProducerFactory"]
    PF --> KP["One shared KafkaProducer"]
    Adm["KafkaAdmin"]
  end
  KP --> Ser["StringSerializer for the key, JsonSerializer for the value"]
  Ser --> Buf["Per-partition batches"]
  Buf --> Snd["Sender thread"]
  Snd --> Topic["Topic orders.placed, 6 partitions, replication factor 3"]
  Topic --> Pay["Payment service consumer group"]
  Topic --> Inv["Inventory service consumer group"]
  Cfg["KafkaTopicConfig declares a NewTopic bean"] --> Adm
  Adm --> Topic
Your code is the controller, the publisher and the topic declaration. Spring Boot builds the template, factory and admin from properties.
  • The application owns three classes: the controller, the publisher and the topic configuration. Everything inside the box comes from auto-configuration and is shaped by spring.kafka.* properties.
  • There is one KafkaProducer for the whole application. All request threads append to the same per-partition batches, and a single sender thread delivers them.
  • KafkaAdmin creates the NewTopic bean at startup if the topic does not exist. Many organisations create topics through their platform tooling instead and set spring.kafka.admin.auto-create=false.
  • The two consumer groups each receive every event. They are outside this lesson, but they are the reason for the key and for publishing without Java type headers.

Message flow

Drawing diagram…

A sequence diagram. The HTTP client posts an order. The controller asks the publisher to publish, the publisher calls KafkaTemplate send and gets a CompletableFuture, and the controller waits on it for up to five seconds. The sender thread sends a Produce request with acks all, the leader acknowledges with partition and offset, the future completes and the controller answers 202. In the failure path the leader reports not enough replicas, the producer retries until the delivery timeout, the future completes exceptionally and the controller answers 503.

sequenceDiagram
  participant H as HTTP client
  participant C as OrderController
  participant P as OrderEventPublisher
  participant T as KafkaTemplate
  participant S as Producer sender thread
  participant L as Leader of orders.placed-2
  H->>C: POST /orders with orderId o-42
  C->>P: publish OrderPlaced
  P->>T: send topic, key o-42, event
  T-->>P: CompletableFuture of SendResult
  P-->>C: the same future
  Note over C: get with a 5 second timeout
  S->>L: Produce request, acks=all
  L-->>S: acknowledged, partition 2 offset 1287
  S-->>C: future completes with SendResult
  C-->>H: 202 Accepted with partition and offset
  Note over S,L: failure path
  S->>L: Produce request
  L-->>S: NOT_ENOUGH_REPLICAS, retried until delivery.timeout.ms
  S-->>C: future completes exceptionally
  C-->>H: 503 Service Unavailable, retry with the same orderId
The request thread waits for the acknowledgement; success and failure both become an explicit HTTP answer.
  1. The future is returned before anything is sent. The controller turns the asynchronous send into a synchronous answer by waiting on it with a timeout.
  2. On success the RecordMetadata gives the partition and offset. Returning or logging them makes it possible to find the exact record later.
  3. In the failure path the producer retries on its own until delivery.timeout.ms. The application does not write a retry loop; it handles the final result.
  4. The 503 tells the client the order was not confirmed. Because the client supplied the order id, retrying is safe for consumers that deduplicate on it.

In a real system

A typical online shop: the order service publishes OrderPlaced to orders.placed with 6 partitions, replication factor 3 and min.insync.replicas=2. The key is the order id. The payment service reads the topic in the group payment-service and the inventory service in the group inventory-service; each group gets every event and can use up to six consumers in parallel.

Traffic is a few hundred orders a second at peak. At that rate throughput tuning is not the concern; correctness and bounded latency are. linger.ms=10 adds at most 10 ms to a checkout that already takes far longer, and lz4 costs little. A clickstream topic in the same company would make the opposite choices, as the previous lesson shows.

Outbox pattern

Simulation, not a real cluster

The animated Build lesson shows the dual-write failure and the outbox that removes it.

Step 1 · The pattern

Save it, then tell everyone

An order comes in. The service saves it to its database, then publishes an event so shipping, email and analytics can react. Two writes, to two systems.

  • order request
  • done
Open the full animated lesson

Java and Spring Boot

The complete service: build file, event, configuration, publisher, controller and a test against an embedded broker. The code was written against the Spring for Apache Kafka 3.3 and Spring Boot 3.5 references and has not been compiled in this environment.

pom.xmlXML
<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0"         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">  <modelVersion>4.0.0</modelVersion>  <parent>    <groupId>org.springframework.boot</groupId>    <artifactId>spring-boot-starter-parent</artifactId>    <version>3.5.16</version>    <relativePath/>  </parent>  <groupId>com.example</groupId>  <artifactId>order-service</artifactId>  <version>0.0.1-SNAPSHOT</version>  <properties>    <java.version>21</java.version>  </properties>  <dependencies>    <dependency>      <groupId>org.springframework.boot</groupId>      <artifactId>spring-boot-starter-web</artifactId>    </dependency>    <dependency>      <groupId>org.springframework.kafka</groupId>      <artifactId>spring-kafka</artifactId>    </dependency>    <dependency>      <groupId>org.springframework.boot</groupId>      <artifactId>spring-boot-starter-test</artifactId>      <scope>test</scope>    </dependency>    <dependency>      <groupId>org.springframework.kafka</groupId>      <artifactId>spring-kafka-test</artifactId>      <scope>test</scope>    </dependency>  </dependencies>  <build>    <plugins>      <plugin>        <groupId>org.springframework.boot</groupId>        <artifactId>spring-boot-maven-plugin</artifactId>      </plugin>    </plugins>  </build></project>

Why it is written this way

  • No versions on the Kafka dependencies. The Spring Boot parent manages spring-kafka (3.3.x) and the matching kafka-clients (3.9.x). Overriding one of them on its own is a common source of class-loading errors.
  • There is no Spring Boot starter for Kafka; spring-kafka on the classpath is what switches the auto-configuration on.
  • spring-boot-starter-web brings Jackson, which JsonSerializer uses.
  • spring-kafka-test provides the embedded broker and KafkaTestUtils used by the test.

Configuration

Spring Boot keys are from the Spring Boot 3.5 application properties reference; where Boot documents no default, the Kafka client's default applies. Kafka client defaults are the documented ones and are the same in the 3.9 and 4.x clients unless stated.

spring.kafka.bootstrap-serversDefault none documented (Spring Boot falls back to localhost:9092)
What it does
Host and port pairs for the initial connection. Applies to producer, consumer and admin unless overridden per client type.
When to change it
List more than one broker so startup does not depend on a single one being up.
spring.kafka.producer.acksDefault none in Spring Boot; Kafka client default all
What it does
How many acknowledgements the leader needs before a request is complete: 0, 1 or all (the in-sync replicas at that moment).
When to change it
Keep all for business events. It is only a durability guarantee together with replication factor 3 and min.insync.replicas=2 on the topic.
spring.kafka.producer.properties.enable.idempotenceDefault true (Kafka client)
What it does
The broker discards duplicates caused by the producer's own retries and keeps batches in order, per partition, within a producer session. Requires acks=all, retries above 0 and at most 5 in-flight requests.
When to change it
Leave it on. Setting it explicitly turns a conflicting setting into a startup error instead of silently disabling it.
spring.kafka.producer.properties.linger.msDefault 0 in kafka-clients 3.9 (Spring Boot 3.5); 5 in kafka-clients 4.x
What it does
How long a batch may wait for more records before it is sent.
When to change it
Set it explicitly. A few milliseconds where a caller waits for the acknowledgement; tens of milliseconds for high-volume streams nobody waits on.
spring.kafka.producer.batch-sizeDefault none in Spring Boot; Kafka client default 16384
What it does
Upper bound in bytes for one partition's batch.
When to change it
Raise it only when batches already fill; see the previous lesson for how to tell.
spring.kafka.producer.compression-typeDefault none in Spring Boot; Kafka client default none
What it does
Codec applied to each batch by the producer: none, gzip, snappy, lz4 or zstd.
When to change it
Enable lz4 or zstd for JSON at any real volume. It changes cost, not guarantees.
spring.kafka.producer.properties.delivery.timeout.msDefault 120000 (2 minutes)
What it does
Upper bound from send returning until success or failure is reported, including time in the buffer and all retries.
When to change it
Lower it when a caller is waiting, but leave room for a leader election. Must be at least linger.ms plus request.timeout.ms.
spring.kafka.producer.properties.max.block.msDefault 60000 (1 minute)
What it does
How long send itself may block waiting for metadata or buffer space.
When to change it
Lower it whenever send is called from a request thread.
spring.kafka.producer.retriesDefault none in Spring Boot; Kafka client default 2147483647
What it does
Number of retries for transient errors. Retries stop at delivery.timeout.ms whatever the count.
When to change it
Leave it unset and control retrying with delivery.timeout.ms, as the Kafka documentation advises. Setting 0 conflicts with idempotence.
spring.kafka.producer.properties.spring.json.add.type.headersDefault true
What it does
Whether JsonSerializer adds record headers naming the Java type of the value.
When to change it
Set false when consumers are other services, or map logical names with spring.json.type.mapping, so consumers do not depend on your class names.
spring.kafka.admin.auto-createDefault true
What it does
Whether NewTopic beans are created on the cluster when the application context starts.
When to change it
Set false where topics are created by platform tooling and applications lack the permission to create them.
spring.kafka.template.observation-enabledDefault false
What it does
Whether KafkaTemplate records Micrometer observations (timers and tracing spans) for sends.
When to change it
Enable it when you run Micrometer tracing so a send appears in the trace of the request that caused it.

Hands-on exercise

Change the service's settings and classify the effect

Start from this service's configuration in the simulator, change one setting at a time and decide for each change whether it moved throughput and latency, or the delivery guarantee.

Steps · 0 of 8 done

Producer configuration simulator: the order service

Simulation · not a real cluster

The preset matches this lesson's application.yml. Change one control at a time and read all three result panels after each change. The simulation is deterministic and conceptual, not a benchmark.

Take it further in the lab: Producer publishing messages

What can go wrong

Ignoring the future returned by send
Symptom
Orders are accepted over HTTP but some never reach downstream services. The only trace is an ERROR line from LoggingProducerListener minutes later.
Cause
send returns before the record leaves the process. The code answered the caller without looking at the outcome, so a failed delivery had nowhere to go.
Fix
Decide per event type: wait on the future with a timeout and fail the request, or attach a callback that records the failure somewhere it will be acted on. For events tied to a database change, use an outbox.
Treating a timeout as “not sent”
Symptom
After a slow period, some orders have two OrderPlaced events with different event ids.
Cause
future.get timed out, the caller retried and the application sent a second record, while the producer went on to deliver the first. Producer idempotence does not cover a second call to send.
Fix
Give events a stable business identity (the client-supplied order id here) and make consumers deduplicate on it. Keep the wait in get and delivery.timeout.ms close to each other so the unknown window is short.
Slow work in the send callback
Symptom
Produce latency climbs for the whole application and record-queue-time-avg grows, although the brokers are healthy.
Cause
A whenComplete callback or ProducerListener writes to a database or calls another service. It runs on the producer's single I/O thread and holds up every batch.
Fix
Keep callbacks to logging and counters, or hand off with whenCompleteAsync and your own executor.
Creating a producer for every request
Symptom
Rising connection counts and memory on brokers, slow sends, no batching, although message volume is ordinary.
Cause
Code builds a new KafkaProducer, DefaultKafkaProducerFactory or KafkaTemplate over a new factory per request or per tenant. Each one opens connections, fetches metadata and registers its own idempotent producer state on the brokers.
Fix
Inject the single auto-configured KafkaTemplate. The Break card A producer for every request shows where this ends.
Saving to the database and sending in the same method
Symptom
Occasionally an order exists in the database with no event, or an event exists for an order that was rolled back.
Cause
Two independent writes. @Transactional on the method covers the database only; the Kafka send is not part of that transaction, and a Kafka transaction would not include the database either.
Fix
Write the event to an outbox table in the same database transaction and publish from there. See transactional outbox.
Type headers tying consumers to the producer's classes
Symptom
A consumer in another service fails to deserialise with an error about a class not being in the trusted packages, or a class that does not exist there.
Cause
JsonSerializer adds type headers by default, naming com.example.orders.events.OrderPlaced. The consuming JsonDeserializer tries to use that class name.
Fix
Publish without type headers and let each consumer set spring.json.value.default.type, or agree on logical names with spring.json.type.mapping on both sides.

How to troubleshoot

The first request hangs for max.block.ms and then fails with a TimeoutException saying the topic is not present in metadata

Check

bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders.placed
How to read it
If the command cannot connect, bootstrap-servers or the network is wrong. If the topic is missing, it was never created: check spring.kafka.admin.auto-create and whether the application is allowed to create topics.
Sends fail with NotEnoughReplicasException after delivery.timeout.ms

Check

bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders.placed (compare the Isr column with min.insync.replicas)
How to read it
Fewer replicas are in sync than min.insync.replicas, so the leader refuses acks=all writes. That is the guarantee working. Fix the brokers; do not lower acks. On a single-broker development cluster, the topic's replication factor and min.insync.replicas must both be 1.
The application fails at startup with a ConfigException about idempotence

Check

The producer section of application.yml: `acks`, `retries` and `max.in.flight.requests.per.connection`
How to read it
Idempotence was enabled explicitly and another setting conflicts with it: acks other than all, retries of 0, or more than 5 in-flight requests. Remove the conflicting setting.
Not sure which settings the producer is actually running with

Check

The `ProducerConfig values:` block that the Kafka client logs at INFO when the first producer is created
How to read it
It lists every effective setting after Spring Boot properties, properties.* overrides and client defaults are merged. Check linger.ms, acks and enable.idempotence there instead of assuming.
Need to see whether sends are failing or being retried in a running service

Check

Producer metrics `record-error-rate`, `record-retry-rate` and `request-latency-avg` (Spring Boot exposes Kafka client metrics through Micrometer when Actuator is present)
How to read it
A retry rate above zero with no errors means the producer is absorbing transient problems. A rising error rate means sends are reaching delivery.timeout.ms or being rejected; look at the exception in the publisher's log.

Key takeaways

  • Spring Boot builds the ProducerFactory, one shared producer and the KafkaTemplate from spring.kafka.*; your code supplies the key, the event and the decision about failure.
  • send returns a future before anything is sent. Someone must look at the outcome, with a timeout if they wait.
  • A timeout means unknown, not failed. Give events a stable business id so that a repeated request is harmless for consumers that deduplicate.
  • Producer idempotence covers the producer's own retries only. It does not cover your application calling send twice.
  • Defaults come from the client library version in the application: Spring Boot 3.5 runs a 3.9 client where linger.ms is 0. Write down the settings you rely on.
  • acks, idempotence and retries set the guarantee; linger.ms, batch.size and compression set cost and latency. Tune the second group freely and the first group deliberately.
  • When an event must match a database write, a direct send is a dual write. Use a transactional outbox.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

Your order service waits on the send future for 5 seconds with delivery.timeout.ms=30000. During a leader election, 40 requests get a timeout from get() and return 503. The mobile app retries each one. Next morning the payment team reports 31 orders with two OrderPlaced events.

What is the most appropriate correction?

Scenario 2 of 3

A colleague proposes this producer section to speed up checkout: acks: 1, enable.idempotence: true, linger.ms: 0, compression-type: none. Load is about 300 orders a second on 6 partitions, the topic has replication factor 3 and min.insync.replicas=2, and p99 produce latency today is 12 ms.

What should you tell them?

Scenario 3 of 3

An order service method annotated @Transactional saves the order with JPA and then calls kafkaTemplate.send(...).get(5, SECONDS). In a load test with brokers restarted at random, 0.2% of orders are in the database with no event, and a handful of events refer to orders that are not in the database.

Which change addresses both symptoms?

References

References: Spring for Apache Kafka 3.3 reference: Sending Messages (KafkaTemplate, ProducerListener, DefaultKafkaProducerFactory) ↗ · Spring for Apache Kafka 3.3 reference: Serialization, Deserialization, and Message Conversion (JsonSerializer, type headers) ↗ · Spring for Apache Kafka 3.3 reference: Testing Applications (@EmbeddedKafka, KafkaTestUtils) ↗ · Spring Boot 3.5 reference: Apache Kafka Support (auto-configuration, additional properties, embedded Kafka) ↗ · Spring Boot 3.5: Common application properties (spring.kafka.producer.*) ↗ · Spring Boot 3.5: Managed dependency coordinates (spring-kafka 3.3.x, kafka-clients 3.9.x) ↗ · Apache Kafka 4.4: Producer configs (acks, enable.idempotence, linger.ms, delivery.timeout.ms, max.block.ms) ↗ · Apache Kafka 3.9: Producer configs (defaults of the client Spring Boot 3.5 manages) ↗ · Apache Kafka 4.4: KafkaProducer Javadoc (callbacks run on the I/O thread) ↗ · Report a problem