Skip to content

Path 9 · lesson 5 of 6

Advanced25 minJava 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 · kafka-clients 3.9 (the versions Spring Boot 3.5 manages: spring-kafka 3.3.x, kafka-clients 3.9.x) against Apache Kafka 4.x brokers. Broker defaults read from the Kafka 4.1 configuration reference.

Kafka transactions in Spring Boot

What transaction-id-prefix switches on, the three ways Spring starts a Kafka transaction, and exactly what happens when a database transaction is involved as well.

What you will learn

  • Say what Spring Boot configures when spring.kafka.producer.transaction-id-prefix is set, and what stops working until every send runs in a transaction
  • Start a Kafka transaction in each of the three ways: executeInTransaction, @Transactional with a KafkaTransactionManager, and the listener container
  • Build a consume-process-produce listener whose output records and input offsets commit together, and configure what happens after a rollback
  • Set read_committed where it is needed and explain what a consumer sees without it
  • Describe, for a listener that also writes to a database, what Spring synchronises, the order of the two commits and the failure window that remains
  • Decide when a Kafka transaction is the right tool and when an outbox is

Before you start

The concept

The lesson Kafka transactions explains the protocol: a transactional producer, a transaction coordinator, commit markers, and offsets sent through the producer so that output and input position move together. This lesson is about the Spring side only: which property turns it on, which object starts and ends the transaction in each situation, and what Spring does when a database transaction is open at the same time.

Everything starts with one property. When spring.kafka.producer.transaction-id-prefix has a value, Spring Boot makes the auto-configured producer factory transactional, creates a KafkaTransactionManager bean, and wires that manager into the listener container factory. From that moment the KafkaTemplate is transactional too, and a transactional template refuses to send outside a transaction.

Who starts the transactionHowWhat is inside itUse it for
Your code, locallykafkaTemplate.executeInTransaction(ops -> ...)The sends made through ops in the callbackPublishing several records atomically from code that is not a listener
Spring's transaction interceptor@Transactional("kafkaTransactionManager") or a TransactionTemplateEvery KafkaTemplate send on that thread until the method returnsThe same, in services that already use declarative transactions
The listener containerA KafkaTransactionManager on the container (Boot wires it)Every send the listener makes and the offset of the consumed recordConsume-process-produce between topics: exactly-once effect on Kafka state
Three entry points, one mechanism underneath: a transactional producer taken from the factory's cache.

Why it exists

With the plain client, a correct transactional read-process-write loop is a page of code: initTransactions, beginTransaction, the sends, sendOffsetsToTransaction with the consumer's group metadata, commitTransaction, and on any failure abortTransaction followed by seeking the consumer back to where the aborted work started. Forget the seek and the consumer silently skips the records it just rolled back. Forget the group metadata and a zombie instance is not fenced.

Spring puts those steps where they belong. The producer factory owns the transactional producers and their ids. The listener container owns the consumer, so it is the one place that knows which offsets belong to the work just done; it begins the transaction before calling your listener, sends the offsets, commits, and repositions the consumer after a rollback. Your listener is left with business code and kafkaTemplate.send.

The second reason is that most Spring services already have a transaction manager for their database. Without guidance, a @Transactional method that also sends to Kafka does a dual write. Spring's integration gives that situation defined behaviour, a fixed commit order, and an exception when the second commit fails. Knowing that behaviour is what lets you decide, service by service, whether it is good enough or whether you need an outbox.

How it works inside

What the prefix changes

  • Producer factory. Instead of one shared producer, DefaultKafkaProducerFactory keeps a cache of transactional producers. Closing one returns it to the cache. Each producer's transactional.id is the prefix plus a counter (payments-router-pod-0-0, -1, ...).
  • Template. Any KafkaTemplate operation outside a transaction throws IllegalStateException (No transaction is in process), unless you set allowNonTransactional on the template.
  • Transaction manager. Boot registers a KafkaTransactionManager bean named kafkaTransactionManager. It is an ordinary PlatformTransactionManager, so @Transactional and TransactionTemplate can use it.
  • Listener containers. Boot sets that manager on the default container factory. Every @KafkaListener built from it now runs inside a Kafka transaction.
  • Error handling. With a transaction manager on the container no CommonErrorHandler is configured by default, so that an exception rolls the transaction back. What happens next is decided by an AfterRollbackProcessor.

A container-managed transaction, step by step

  1. The container polls. For a record listener it then handles the records one transaction per record; a batch listener gets one transaction per batch.
  2. It asks the KafkaTransactionManager to begin. A transactional producer is taken from the cache, beginTransaction is called, and the producer is bound to the listener thread.
  3. Your listener runs. Each kafkaTemplate.send finds the bound producer and uses it, so the records are written as part of the open transaction.
  4. The listener returns. The container calls sendOffsetsToTransaction with the offset after the consumed record and the consumer's group metadata. The offset commit is now part of the same transaction.
  5. The transaction manager commits. The coordinator writes commit markers to the output partitions and to the offsets partition. Consumers with read_committed can now see the output.
  6. If the listener throws, the transaction is aborted: the output records are marked aborted and the offset does not move. The AfterRollbackProcessor then seeks the consumer so the record is fetched again on the next poll.

The default DefaultAfterRollbackProcessor behaves like the default error handler: after ten failures of the same record it logs it and lets it go. Give it a recoverer and a back-off. If you also pass it the template and set commitRecovered to true, it runs the recovery in a new transaction and sends the failed record's offset to that transaction, so that the dead-letter publish and the move past the record are atomic as well. Non-blocking retries (@RetryableTopic) cannot be combined with container transactions.

Reading: isolation.level

Aborted records are not removed from the log. They stay in the partition and are marked by an abort marker. A consumer with the default isolation.level=read_uncommitted receives them like any other record. Only a consumer with read_committed skips aborted records, and it also reads no further than the first still-open transaction. A rollback therefore protects downstream services only if they set read_committed. Set it on every consumer of a topic that is written transactionally, including the consumer of your own service when its input comes from another transactional stage.

When a database is involved

Spring never merges the two transactions. What it offers is synchronisation: one transaction is the primary one, the other is tied to it and is committed or rolled back right after it. The lesson Kafka transactions and your database derives why nothing stronger is possible; the table below is what the Spring reference states for each arrangement.

ArrangementWhat Spring doesCommit orderIf the second commit fails
Producer only: method is @Transactional on the database manager and calls kafkaTemplate.sendThe template starts a Kafka transaction and registers it with the database transactionDatabase, then KafkaThe row is committed and the event is not. The failure is thrown to the caller (since 2.5.17 / 2.6.12 / 2.7.9 / 2.8.0; earlier it was only logged at debug). Nothing retries.
Producer only, nested: outer method on the database manager calls a method on another bean that is @Transactional("kafkaTransactionManager")The inner method's Kafka transaction commits when it returns, before the outer method endsKafka, then databaseThe event is committed and the row is not: a phantom event.
Listener: the container starts the Kafka transaction, the listener method is @Transactional on the database managerThe database transaction begins and ends inside the listener call; the container then sends offsets and commits KafkaDatabase, then offsets and KafkaThe output is aborted and the record is delivered again, so the database work runs a second time.
Ordered commits, not a distributed transaction. Each row has a window between its two commits.

Only the third row repairs itself, because the input record is still in the log and its offset did not move. That is why the reference's own example adds that the database update should be idempotent. It also means the listener must send its output again on the second run even when the database says the work is already done: the first run's output was aborted.

Crash between the two commits

Simulation · not a real cluster

Transactional mode with a crash after processing. Watch what Kafka rolls back and which side effect has already happened.

Architecture

Drawing diagram…

Flowchart. The transaction id prefix property leads to a transactional producer factory and a Kafka transaction manager. The template, the listener container and transactional methods all end up using a transactional producer bound to the thread, which writes to output partitions. The listener container also sends offsets. Both go to the transaction coordinator. A separate database transaction manager writes database rows and is linked to the Kafka producer only by commit ordering.

flowchart TD
  PROP["spring.kafka.producer.transaction-id-prefix"] --> PF["Producer factory with a cache of transactional producers"]
  PROP --> KTM["KafkaTransactionManager bean"]
  PF --> KT["KafkaTemplate, sends only inside a transaction"]
  KTM --> LC["Listener container starts a transaction per record"]
  KTM --> ANN["Transactional methods naming kafkaTransactionManager"]
  KT --> LOCAL["executeInTransaction callback"]
  LC --> TXP["Transactional producer bound to the thread"]
  ANN --> TXP
  LOCAL --> TXP
  TXP --> OUT["Output topic partitions"]
  LC --> OFF["Offsets of the consumed record"]
  OFF --> TC["Transaction coordinator on a broker"]
  OUT --> TC
  DBTM["Database transaction manager, primary"] --> DB["Database rows"]
  DBTM -.->|commit order only| TXP
What one property creates, the three ways a transaction is started, and the database manager standing outside it.
  • All three entry points reach the same thing: a producer with a transactional.id, bound to the current thread for the length of the transaction.
  • Only the listener container adds consumer offsets, because only it owns the consumer. executeInTransaction and a @Transactional service method give you atomic writes, not consume-process-produce.
  • The transaction coordinator sees output partitions and the offsets partition. It never sees the database.
  • The dotted line is the entire relationship between the database manager and Kafka: Spring calls one commit and then the other.

Message flow

Drawing diagram…

Sequence diagram. The container polls a record, begins a Kafka transaction and invokes the listener. The listener writes to the database, sends an output record and the database commits. After the listener returns, the container sends the offset to the transaction and commits it. A note marks the window between the two commits. In the failure branch the Kafka transaction aborts, the container seeks back and the listener is invoked again for the same record.

sequenceDiagram
  participant K as Broker and coordinator
  participant C as Listener container
  participant L as StockReservationListener
  participant D as Inventory DB
  C->>K: poll
  K-->>C: reservation-requested for order 7001, offset 88
  C->>K: begin Kafka transaction
  C->>L: invoke listener
  L->>D: BEGIN, insert reservation, update stock
  L->>K: send stock-reserved for order 7001
  L->>D: COMMIT
  D-->>L: committed
  L-->>C: return
  Note over C,D: failure window, the database is committed and Kafka is not
  C->>K: send offset 89 to the transaction
  C->>K: commit Kafka transaction
  K-->>C: committed, stock-reserved becomes visible
  Note over K,D: if the Kafka commit fails instead
  K--xC: abort, stock-reserved is discarded
  C->>K: seek to offset 88 and poll
  K-->>C: reservation-requested for order 7001 again
  C->>L: invoke listener a second time
A listener with a database transaction inside a container-managed Kafka transaction: database first, Kafka second.
  1. The output record is sent before the database commit, but it is not visible to read_committed consumers until the Kafka commit at the end.
  2. The database commit happens when the listener method returns, because @Transactional wraps the method. The Kafka commit happens afterwards, in the container.
  3. In the window between them, a crash or a failed Kafka commit leaves the reservation in the database with no stock-reserved event and no offset movement.
  4. The second invocation must find the existing reservation, change nothing, and still send stock-reserved. Then both sides agree.

In a real system

Two services from the example project show where a Kafka transaction is enough and where it is not.

  • Payment result router (Kafka in, Kafka out). It reads payments.payment-result (12 partitions, keyed by orderId) in group payment-result-router. For an approved payment it writes orders.order-confirmed; for a declined one it writes orders.order-cancelled; in both cases it writes an audit record to payments.audit. No database. A container-managed transaction makes the two output records and the input offset atomic, so a crash can never leave an audit entry without a decision or produce the decision twice for committed readers.
  • Inventory service (Kafka in, database, Kafka out). It reads inventory.reservation-requested, decrements stock in PostgreSQL and publishes inventory.stock-reserved. Here the Kafka transaction covers the output record and the offset, and the database is outside. The service makes the reservation idempotent with a unique key on order_id, so the redelivery that follows a failed Kafka commit is harmless.

The order service, which only produces (INSERT an order, publish OrderPlaced), uses neither arrangement. It has no input record to be redelivered, so a failed second commit would never be repaired. It writes an outbox row instead. For the inventory service the outbox is also the more common choice in practice: one database transaction, no transactional producer to operate, and the same idempotency requirement it already has.

Java and Spring Boot

Three pieces: the Boot configuration that makes a service transactional, the Kafka-only router with its after-rollback processor, and the inventory listener that adds a database transaction. Topic and record names match the example project at /kafka/project.

payment-result-router/src/main/resources/application.ymlYAML
spring:  application:    name: payment-result-router  kafka:    bootstrap-servers: localhost:9092    producer:      # Makes the producer factory transactional. Must be different on every running instance.      transaction-id-prefix: ${spring.application.name}-${HOSTNAME}-      key-serializer: org.apache.kafka.common.serialization.StringSerializer      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer      properties:        "[transaction.timeout.ms]": 30000    consumer:      group-id: payment-result-router      auto-offset-reset: earliest      isolation-level: read-committed      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer      properties:        "[spring.json.value.default.type]": com.example.router.PaymentResult        "[spring.json.use.type.headers]": false        "[spring.json.trusted.packages]": com.example.router

Why it is written this way

  • transaction-id-prefix is the only switch. With it, Boot also creates the KafkaTransactionManager and sets it on the default listener container factory.
  • HOSTNAME is the pod name in Kubernetes. In a StatefulSet it is stable across restarts, so a restarted instance reuses its transactional.id values and the coordinator aborts whatever its predecessor left open. With names that change on every restart, an abandoned transaction is aborted only when transaction.timeout.ms expires.
  • isolation-level: read-committed is on this consumer because payments.payment-result may itself be written in a transaction by the payment service. The services that read this router's output need the same setting.
  • transaction.timeout.ms is lowered from the default of one minute. While a transaction is open, read_committed consumers of the output partitions cannot read past it, so a hung instance should be cleaned up quickly. Keep it above the longest time one listener call can take.
  • A transactional producer requires idempotence and acks=all. Both are the client defaults; do not override them.
  • The deserializers are shown unwrapped to keep the file short. Use ErrorHandlingDeserializer as in the previous lesson.

Configuration

Spring Boot properties first, then the Kafka settings they depend on. Client defaults are those of kafka-clients 3.9 (managed by Spring Boot 3.5); broker defaults are from the Kafka 4.1 reference and are the same in 3.9.

spring.kafka.producer.transaction-id-prefixDefault none (transactions off)
What it does
Makes the auto-configured producer factory transactional. Boot then also creates a KafkaTransactionManager and sets it on the default listener container factory.
When to change it
Set it only in services that need atomic writes or consume-process-produce. It must be unique per application instance, and it makes every send require a transaction.
spring.kafka.consumer.isolation-levelDefault read-uncommitted
What it does
Maps to the consumer's isolation.level. With read-committed the consumer returns transactional records only after their transaction has committed.
When to change it
Set read-committed on every consumer of a transactionally written topic. With the default, consumers also process records of aborted transactions.
transaction.timeout.ms (producer)Default 60000 (1 minute)
What it does
How long the coordinator waits for a transaction to finish before it aborts it.
When to change it
Lower it so abandoned transactions stop blocking read_committed readers sooner, but keep it above your slowest listener call. Set it through spring.kafka.producer.properties.
transaction.max.timeout.ms (broker)Default 900000 (15 minutes)
What it does
The largest transaction.timeout.ms a broker accepts from a producer.
When to change it
A producer asking for more is rejected when it initialises. Needing more is a sign that slow work sits inside the transaction.
transactional.id.expiration.ms (broker)Default 604800000 (7 days)
What it does
How long the coordinator remembers a transactional.id that has sent no transaction status updates.
When to change it
A producer idle for longer is rejected on its next use. For rarely used producers, set maxAge on DefaultKafkaProducerFactory below this value so that cached producers are refreshed.
transaction.state.log.replication.factor / transaction.state.log.min.isr (broker)Default 3 / 2
What it does
Replication of the internal transaction state topic. The topic is created on first use.
When to change it
Leave them alone in production. On a single-broker development cluster transactions cannot start until both are set to 1; the Testcontainers Kafka module does that for you.
KafkaTemplate.allowNonTransactionalDefault false
What it does
Whether a transactional template may send outside a transaction, using a separate non-transactional producer.
When to change it
Prefer a second, non-transactional producer factory and template for sends that do not belong in a transaction. Mixing both through one template hides which sends are atomic.
spring.kafka.listener.typeDefault single
What it does
Record listener or batch listener. In a transactional container it also sets the size of a transaction: one record or one poll.
When to change it
Use batch for high-volume stages to cut the per-record transaction cost. A failure then rolls back and redelivers the whole batch.
ContainerProperties.stopContainerWhenFencedDefault false
What it does
Stops the container when its producer is fenced, instead of carrying on.
When to change it
The container cannot tell a fencing caused by a rebalance from one caused by a timed-out or expired transactional.id. Enable it if a silent skip after expiry would be worse than a stopped listener, and alert on the stop.

Hands-on exercise

Trace the two commits, then break them in the lab

Predict what each topic, the consumer group and the database contain after a failure at three different points of a transactional listener, and confirm the Kafka side in the transactions lab.

Steps · 0 of 7 done

Transactional listener with an idempotent database write

Simulation · not a real cluster

Crash after processing, before the commit. See which effects Kafka undoes and why the repeated database write is harmless here.

Take it further in the lab: Kafka transaction

What can go wrong

Sends that fail after the prefix was added
Symptom
A REST endpoint that publishes an event starts returning 500 with IllegalStateException: No transaction is in process right after transactions were enabled for a listener.
Cause
The prefix makes the shared KafkaTemplate transactional, and a transactional template refuses to send outside a transaction.
Fix
Wrap such sends in executeInTransaction or a @Transactional("kafkaTransactionManager") method, or give that code path its own non-transactional producer factory and template.
The same prefix on every instance
Symptom
With two or more replicas, listeners log ProducerFencedException over and over and throughput collapses; one replica alone works.
Cause
All instances generate the same transactional.id values (prefix plus counter). Each new producer fences the one with the same id on another instance.
Fix
Include a per-instance value, such as the pod name, in transaction-id-prefix.
Rollbacks that downstream never notices
Symptom
The order service acts on confirmations that the router aborted. The router's logs show Transaction rolled back for exactly those records.
Cause
The order service's consumer uses the default isolation.level=read_uncommitted, which returns aborted records too.
Fix
Set spring.kafka.consumer.isolation-level=read-committed in every service that consumes transactionally written topics.
@Transactional on the wrong manager
Symptom
After adding the prefix, JPA or JDBC changes in a @Transactional listener are no longer rolled back on an exception, or the application fails to start with more than one transaction manager candidate.
Cause
Two PlatformTransactionManager beans exist, or the database one is no longer auto-configured, and the annotation does not say which to use.
Fix
Declare the database transaction manager explicitly as @Primary and name the manager in @Transactional on listeners and Kafka-only methods.
Skipping the send on a duplicate
Symptom
A reservation exists in the database but inventory.stock-reserved was never published; the order waits for ever. It happened during a broker restart.
Cause
The database committed, the Kafka commit failed, and on redelivery the listener saw the existing row and returned early without sending. The first send had been aborted.
Fix
On a detected duplicate, skip the database change but still produce the output, with the same event id. Or move the output to an outbox row written with the reservation.
Slow work inside the transaction
Symptom
Intermittent aborted transactions and fenced producers under load; consumers of the output topic show lag that clears in bursts.
Cause
An HTTP call or a slow query inside the listener keeps the Kafka transaction open past transaction.timeout.ms, and open transactions hold back read_committed readers.
Fix
Keep remote calls out of transactional listeners. Do the slow step in a separate, idempotent stage and keep transactions short.

How to troubleshoot

A read_committed consumer of the output topic stops although the producer is writing

Check

kafka-transactions.sh --bootstrap-server localhost:9092 list, then kafka-transactions.sh --bootstrap-server localhost:9092 describe --transactional-id payment-result-router-pod-0-0
How to read it
A transaction in state Ongoing with an old start time is holding back the last stable offset. Its producer hung or died; it is aborted when transaction.timeout.ms expires, or when the same transactional.id initialises again.
You need to know whether a record was committed or aborted

Check

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.order-confirmed --from-beginning --isolation-level read_committed
How to read it
Run it again without the isolation option (the default is read_uncommitted). A record that shows only in the second run belongs to an aborted or still-open transaction.
The application hangs or fails at the first transactional send on a local single broker

Check

Producer log: a TimeoutException while awaiting InitProducerId; broker settings transaction.state.log.replication.factor and transaction.state.log.min.isr
How to read it
The internal transaction state topic cannot be created with replication factor 3 on one broker. Set both settings to 1 for local development only.
The same record is processed again and again in a transactional listener

Check

Service log at ERROR: `Transaction rolled back`, followed by the listener's exception; then the after-rollback processor's configuration
How to read it
Each rollback seeks back to the record. Without a recoverer it is retried ten times and then only logged. Configure a DefaultAfterRollbackProcessor with a dead-letter recoverer.
Offsets of the group do not move even though output is produced

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-result-router
How to read it
In a transactional container offsets are committed through the producer as part of the transaction. If CURRENT-OFFSET stands still while output appears for read_uncommitted readers, the transactions are not committing: look for commit failures or fencing in the log.

Key takeaways

  • spring.kafka.producer.transaction-id-prefix makes the producer factory and template transactional, adds a KafkaTransactionManager, and puts every default listener container in a transaction.
  • Use executeInTransaction or @Transactional("kafkaTransactionManager") for atomic writes; only the listener container adds the consumed offsets and gives consume-process-produce.
  • In a transactional container the AfterRollbackProcessor replaces the error handler. Configure it with a recoverer, a back-off and commitRecovered.
  • A rollback protects only consumers that read with isolation.level=read_committed.
  • With a database, Spring commits the database first and Kafka second. A failure in between redelivers the record on the consumer side and is repaired by nothing on the producer side.
  • Make the database write idempotent and still send the output on a redelivery, or use an outbox, which is usually the simpler design when a database is involved.
  • Give every instance its own prefix, keep transactions short, and declare the database transaction manager explicitly.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A team sets spring.kafka.producer.transaction-id-prefix: router- in a service with 3 replicas. A single replica in the test environment works. In production the logs fill with ProducerFencedException, listeners keep rolling back and lag grows on all 12 partitions.

What should they change first?

Scenario 2 of 3

An inventory listener runs in a container-managed Kafka transaction and is @Transactional on the JDBC manager. It inserts a reservation (unique on order_id), returns early if the row already exists, and otherwise sends stock-reserved. After a broker restart, 6 orders have a reservation row but no stock-reserved event, and their offsets have been committed.

Which change fixes the cause?

Scenario 3 of 3

An order service handles POST /orders: it inserts an order with JPA and must publish OrderPlaced. A developer proposes setting a transaction-id-prefix so the @Transactional method commits the database and then Kafka. Load is 200 orders per second and a lost event means an order that is never fulfilled.

What is the stronger design for this service?

References

References: Spring for Apache Kafka 3.3 reference: Transactions (overview, KafkaTransactionManager, synchronization, local transactions, transactionIdPrefix) ↗ · Spring for Apache Kafka 3.3 reference: Exactly Once Semantics (container-managed transactions, EOSMode.V2) ↗ · Spring for Apache Kafka 3.3 reference: Examples of Kafka Transactions with Other Transaction Managers ↗ · Spring for Apache Kafka 3.3 reference: Handling Exceptions (After Rollback Processor) ↗ · Spring Boot 3.5 reference: Apache Kafka support (transaction-id-prefix, beans attached to the container factory) ↗ · Spring Boot 3.5: Managed dependency coordinates (spring-kafka 3.3.x, kafka-clients 3.9.x) ↗ · Apache Kafka 3.9 producer configuration reference (transaction.timeout.ms, transactional.id) ↗ · Apache Kafka 4.1 broker configuration reference (transaction.max.timeout.ms, transactional.id.expiration.ms, transaction state log settings) ↗ · KIP-447: Producer scalability for exactly once semantics (fencing by consumer group metadata) ↗ · Report a problem