Path 9 · lesson 5 of 6
Kafka transactions in Spring Boot
What transaction-id-prefix switches on, the three ways Spring starts a Kafka transaction, and exactly what happens when a database transaction is involved as well.
What you will learn
- Say what Spring Boot configures when
spring.kafka.producer.transaction-id-prefixis set, and what stops working until every send runs in a transaction - Start a Kafka transaction in each of the three ways:
executeInTransaction,@Transactionalwith aKafkaTransactionManager, and the listener container - Build a consume-process-produce listener whose output records and input offsets commit together, and configure what happens after a rollback
- Set
read_committedwhere it is needed and explain what a consumer sees without it - Describe, for a listener that also writes to a database, what Spring synchronises, the order of the two commits and the failure window that remains
- Decide when a Kafka transaction is the right tool and when an outbox is
Before you start
- Kafka transactions
- Kafka transactions and your database
- Spring Kafka setup
- Error handling, retries and dead letters in Spring
- Spring's @Transactional and PlatformTransactionManager
The concept
The lesson Kafka transactions explains the protocol: a transactional producer, a transaction coordinator, commit markers, and offsets sent through the producer so that output and input position move together. This lesson is about the Spring side only: which property turns it on, which object starts and ends the transaction in each situation, and what Spring does when a database transaction is open at the same time.
Everything starts with one property. When spring.kafka.producer.transaction-id-prefix has a value, Spring Boot makes the auto-configured producer factory transactional, creates a KafkaTransactionManager bean, and wires that manager into the listener container factory. From that moment the KafkaTemplate is transactional too, and a transactional template refuses to send outside a transaction.
| Who starts the transaction | How | What is inside it | Use it for |
|---|---|---|---|
| Your code, locally | kafkaTemplate.executeInTransaction(ops -> ...) | The sends made through ops in the callback | Publishing several records atomically from code that is not a listener |
| Spring's transaction interceptor | @Transactional("kafkaTransactionManager") or a TransactionTemplate | Every KafkaTemplate send on that thread until the method returns | The same, in services that already use declarative transactions |
| The listener container | A KafkaTransactionManager on the container (Boot wires it) | Every send the listener makes and the offset of the consumed record | Consume-process-produce between topics: exactly-once effect on Kafka state |
Why it exists
With the plain client, a correct transactional read-process-write loop is a page of code: initTransactions, beginTransaction, the sends, sendOffsetsToTransaction with the consumer's group metadata, commitTransaction, and on any failure abortTransaction followed by seeking the consumer back to where the aborted work started. Forget the seek and the consumer silently skips the records it just rolled back. Forget the group metadata and a zombie instance is not fenced.
Spring puts those steps where they belong. The producer factory owns the transactional producers and their ids. The listener container owns the consumer, so it is the one place that knows which offsets belong to the work just done; it begins the transaction before calling your listener, sends the offsets, commits, and repositions the consumer after a rollback. Your listener is left with business code and kafkaTemplate.send.
The second reason is that most Spring services already have a transaction manager for their database. Without guidance, a @Transactional method that also sends to Kafka does a dual write. Spring's integration gives that situation defined behaviour, a fixed commit order, and an exception when the second commit fails. Knowing that behaviour is what lets you decide, service by service, whether it is good enough or whether you need an outbox.
How it works inside
What the prefix changes
- Producer factory. Instead of one shared producer,
DefaultKafkaProducerFactorykeeps a cache of transactional producers. Closing one returns it to the cache. Each producer'stransactional.idis the prefix plus a counter (payments-router-pod-0-0,-1, ...). - Template. Any
KafkaTemplateoperation outside a transaction throwsIllegalStateException(No transaction is in process), unless you setallowNonTransactionalon the template. - Transaction manager. Boot registers a
KafkaTransactionManagerbean namedkafkaTransactionManager. It is an ordinaryPlatformTransactionManager, so@TransactionalandTransactionTemplatecan use it. - Listener containers. Boot sets that manager on the default container factory. Every
@KafkaListenerbuilt from it now runs inside a Kafka transaction. - Error handling. With a transaction manager on the container no
CommonErrorHandleris configured by default, so that an exception rolls the transaction back. What happens next is decided by anAfterRollbackProcessor.
A container-managed transaction, step by step
- The container polls. For a record listener it then handles the records one transaction per record; a batch listener gets one transaction per batch.
- It asks the
KafkaTransactionManagerto begin. A transactional producer is taken from the cache,beginTransactionis called, and the producer is bound to the listener thread. - Your listener runs. Each
kafkaTemplate.sendfinds the bound producer and uses it, so the records are written as part of the open transaction. - The listener returns. The container calls
sendOffsetsToTransactionwith the offset after the consumed record and the consumer's group metadata. The offset commit is now part of the same transaction. - The transaction manager commits. The coordinator writes commit markers to the output partitions and to the offsets partition. Consumers with
read_committedcan now see the output. - If the listener throws, the transaction is aborted: the output records are marked aborted and the offset does not move. The
AfterRollbackProcessorthen seeks the consumer so the record is fetched again on the next poll.
The default DefaultAfterRollbackProcessor behaves like the default error handler: after ten failures of the same record it logs it and lets it go. Give it a recoverer and a back-off. If you also pass it the template and set commitRecovered to true, it runs the recovery in a new transaction and sends the failed record's offset to that transaction, so that the dead-letter publish and the move past the record are atomic as well. Non-blocking retries (@RetryableTopic) cannot be combined with container transactions.
Reading: isolation.level
Aborted records are not removed from the log. They stay in the partition and are marked by an abort marker. A consumer with the default isolation.level=read_uncommitted receives them like any other record. Only a consumer with read_committed skips aborted records, and it also reads no further than the first still-open transaction. A rollback therefore protects downstream services only if they set read_committed. Set it on every consumer of a topic that is written transactionally, including the consumer of your own service when its input comes from another transactional stage.
When a database is involved
Spring never merges the two transactions. What it offers is synchronisation: one transaction is the primary one, the other is tied to it and is committed or rolled back right after it. The lesson Kafka transactions and your database derives why nothing stronger is possible; the table below is what the Spring reference states for each arrangement.
| Arrangement | What Spring does | Commit order | If the second commit fails |
|---|---|---|---|
Producer only: method is @Transactional on the database manager and calls kafkaTemplate.send | The template starts a Kafka transaction and registers it with the database transaction | Database, then Kafka | The row is committed and the event is not. The failure is thrown to the caller (since 2.5.17 / 2.6.12 / 2.7.9 / 2.8.0; earlier it was only logged at debug). Nothing retries. |
Producer only, nested: outer method on the database manager calls a method on another bean that is @Transactional("kafkaTransactionManager") | The inner method's Kafka transaction commits when it returns, before the outer method ends | Kafka, then database | The event is committed and the row is not: a phantom event. |
Listener: the container starts the Kafka transaction, the listener method is @Transactional on the database manager | The database transaction begins and ends inside the listener call; the container then sends offsets and commits Kafka | Database, then offsets and Kafka | The output is aborted and the record is delivered again, so the database work runs a second time. |
Only the third row repairs itself, because the input record is still in the log and its offset did not move. That is why the reference's own example adds that the database update should be idempotent. It also means the listener must send its output again on the second run even when the database says the work is already done: the first run's output was aborted.
Crash between the two commits
Simulation · not a real clusterTransactional mode with a crash after processing. Watch what Kafka rolls back and which side effect has already happened.
Architecture
Drawing diagram…
Flowchart. The transaction id prefix property leads to a transactional producer factory and a Kafka transaction manager. The template, the listener container and transactional methods all end up using a transactional producer bound to the thread, which writes to output partitions. The listener container also sends offsets. Both go to the transaction coordinator. A separate database transaction manager writes database rows and is linked to the Kafka producer only by commit ordering.
flowchart TD PROP["spring.kafka.producer.transaction-id-prefix"] --> PF["Producer factory with a cache of transactional producers"] PROP --> KTM["KafkaTransactionManager bean"] PF --> KT["KafkaTemplate, sends only inside a transaction"] KTM --> LC["Listener container starts a transaction per record"] KTM --> ANN["Transactional methods naming kafkaTransactionManager"] KT --> LOCAL["executeInTransaction callback"] LC --> TXP["Transactional producer bound to the thread"] ANN --> TXP LOCAL --> TXP TXP --> OUT["Output topic partitions"] LC --> OFF["Offsets of the consumed record"] OFF --> TC["Transaction coordinator on a broker"] OUT --> TC DBTM["Database transaction manager, primary"] --> DB["Database rows"] DBTM -.->|commit order only| TXP
- All three entry points reach the same thing: a producer with a
transactional.id, bound to the current thread for the length of the transaction. - Only the listener container adds consumer offsets, because only it owns the consumer.
executeInTransactionand a@Transactionalservice method give you atomic writes, not consume-process-produce. - The transaction coordinator sees output partitions and the offsets partition. It never sees the database.
- The dotted line is the entire relationship between the database manager and Kafka: Spring calls one commit and then the other.
Message flow
Drawing diagram…
Sequence diagram. The container polls a record, begins a Kafka transaction and invokes the listener. The listener writes to the database, sends an output record and the database commits. After the listener returns, the container sends the offset to the transaction and commits it. A note marks the window between the two commits. In the failure branch the Kafka transaction aborts, the container seeks back and the listener is invoked again for the same record.
sequenceDiagram participant K as Broker and coordinator participant C as Listener container participant L as StockReservationListener participant D as Inventory DB C->>K: poll K-->>C: reservation-requested for order 7001, offset 88 C->>K: begin Kafka transaction C->>L: invoke listener L->>D: BEGIN, insert reservation, update stock L->>K: send stock-reserved for order 7001 L->>D: COMMIT D-->>L: committed L-->>C: return Note over C,D: failure window, the database is committed and Kafka is not C->>K: send offset 89 to the transaction C->>K: commit Kafka transaction K-->>C: committed, stock-reserved becomes visible Note over K,D: if the Kafka commit fails instead K--xC: abort, stock-reserved is discarded C->>K: seek to offset 88 and poll K-->>C: reservation-requested for order 7001 again C->>L: invoke listener a second time
- The output record is sent before the database commit, but it is not visible to
read_committedconsumers until the Kafka commit at the end. - The database commit happens when the listener method returns, because
@Transactionalwraps the method. The Kafka commit happens afterwards, in the container. - In the window between them, a crash or a failed Kafka commit leaves the reservation in the database with no
stock-reservedevent and no offset movement. - The second invocation must find the existing reservation, change nothing, and still send
stock-reserved. Then both sides agree.
In a real system
Two services from the example project show where a Kafka transaction is enough and where it is not.
- Payment result router (Kafka in, Kafka out). It reads
payments.payment-result(12 partitions, keyed byorderId) in grouppayment-result-router. For an approved payment it writesorders.order-confirmed; for a declined one it writesorders.order-cancelled; in both cases it writes an audit record topayments.audit. No database. A container-managed transaction makes the two output records and the input offset atomic, so a crash can never leave an audit entry without a decision or produce the decision twice for committed readers. - Inventory service (Kafka in, database, Kafka out). It reads
inventory.reservation-requested, decrements stock in PostgreSQL and publishesinventory.stock-reserved. Here the Kafka transaction covers the output record and the offset, and the database is outside. The service makes the reservation idempotent with a unique key onorder_id, so the redelivery that follows a failed Kafka commit is harmless.
The order service, which only produces (INSERT an order, publish OrderPlaced), uses neither arrangement. It has no input record to be redelivered, so a failed second commit would never be repaired. It writes an outbox row instead. For the inventory service the outbox is also the more common choice in practice: one database transaction, no transactional producer to operate, and the same idempotency requirement it already has.
Java and Spring Boot
Three pieces: the Boot configuration that makes a service transactional, the Kafka-only router with its after-rollback processor, and the inventory listener that adds a database transaction. Topic and record names match the example project at /kafka/project.
spring: application: name: payment-result-router kafka: bootstrap-servers: localhost:9092 producer: # Makes the producer factory transactional. Must be different on every running instance. transaction-id-prefix: ${spring.application.name}-${HOSTNAME}- key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: "[transaction.timeout.ms]": 30000 consumer: group-id: payment-result-router auto-offset-reset: earliest isolation-level: read-committed key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer properties: "[spring.json.value.default.type]": com.example.router.PaymentResult "[spring.json.use.type.headers]": false "[spring.json.trusted.packages]": com.example.routerWhy it is written this way
transaction-id-prefixis the only switch. With it, Boot also creates theKafkaTransactionManagerand sets it on the default listener container factory.HOSTNAMEis the pod name in Kubernetes. In a StatefulSet it is stable across restarts, so a restarted instance reuses itstransactional.idvalues and the coordinator aborts whatever its predecessor left open. With names that change on every restart, an abandoned transaction is aborted only whentransaction.timeout.msexpires.isolation-level: read-committedis on this consumer becausepayments.payment-resultmay itself be written in a transaction by the payment service. The services that read this router's output need the same setting.transaction.timeout.msis lowered from the default of one minute. While a transaction is open,read_committedconsumers of the output partitions cannot read past it, so a hung instance should be cleaned up quickly. Keep it above the longest time one listener call can take.- A transactional producer requires idempotence and
acks=all. Both are the client defaults; do not override them. - The deserializers are shown unwrapped to keep the file short. Use
ErrorHandlingDeserializeras in the previous lesson.
package com.example.router;import java.time.Instant;import java.util.UUID;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.stereotype.Component;record PaymentResult(UUID eventId, UUID paymentId, UUID orderId, boolean approved, String providerRef, Instant occurredAt) {}record OrderConfirmed(UUID eventId, UUID orderId, UUID paymentId) {}record OrderCancelled(UUID eventId, UUID orderId, String reason) {}record PaymentAudit(UUID paymentId, UUID orderId, boolean approved, Instant routedAt) {}@Componentpublic class PaymentResultRouter { private final KafkaTemplate<String, Object> kafka; public PaymentResultRouter(KafkaTemplate<String, Object> kafka) { this.kafka = kafka; } /** * Runs inside the Kafka transaction the container started. No annotation is needed: * both sends and the offset of the consumed record commit or abort together. */ @KafkaListener(id = "payment-result-router", topics = "payments.payment-result") public void route(PaymentResult result) { String key = result.orderId().toString(); if (result.approved()) { kafka.send("orders.order-confirmed", key, new OrderConfirmed(result.eventId(), result.orderId(), result.paymentId())); } else { kafka.send("orders.order-cancelled", key, new OrderCancelled(result.eventId(), result.orderId(), "PAYMENT_DECLINED")); } kafka.send("payments.audit", key, new PaymentAudit(result.paymentId(), result.orderId(), result.approved(), Instant.now())); }}Why it is written this way
- The listener does not begin, commit or acknowledge anything. The container began the transaction before calling it and will send the offset and commit after it returns.
- The sends are not awaited. In a transaction the commit is what tells you they succeeded: if any record cannot be written, the commit fails and everything is aborted.
- The output events reuse the input's
eventId. If this stage is ever replayed, downstream consumers can still recognise the same business event. - This is exactly-once for Kafka state: for consumers reading with
read_committed, each payment result yields its outputs once. Ifroutealso called an HTTP API, that call would happen again on every redelivery.
package com.example.router;import org.apache.kafka.common.TopicPartition;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;import org.springframework.kafka.listener.DefaultAfterRollbackProcessor;import org.springframework.util.backoff.FixedBackOff;@Configurationpublic class KafkaTransactionConfig { /** * Spring Boot attaches an AfterRollbackProcessor bean to the default container factory. * In a transactional container this takes the place of the DefaultErrorHandler. */ @Bean DefaultAfterRollbackProcessor<Object, Object> afterRollbackProcessor(KafkaTemplate<Object, Object> template) { DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template, (record, exception) -> new TopicPartition(record.topic() + "-dlt", record.partition())); // Three deliveries, two seconds apart. Then, in a new transaction, publish the record // to the dead-letter topic and send its offset to that same transaction. return new DefaultAfterRollbackProcessor<>(recoverer, new FixedBackOff(2_000L, 2L), template, true); }}Why it is written this way
- The four-argument constructor is
(recoverer, backOff, kafkaOperations, commitRecovered). WithcommitRecoveredtrue the template is required, and the dead-letter publish and the offset of the failed record are committed in one new transaction. - Without this bean the default processor redelivers a failing record ten times and then only logs it, the same trap as the default error handler.
- A
DefaultErrorHandlerbean would not do this job here. With transactions, a failure has to roll the transaction back first; the after-rollback processor runs after that. - With a batch listener the whole batch is redelivered after a rollback, because the framework does not know which record failed. Recovering a batch has to be enabled separately with
ContainerProperties.setBatchRecoverAfterRollback(true)(since 3.2).
package com.example.inventory;import java.util.List;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.stereotype.Service;import org.springframework.transaction.annotation.Transactional;record StockAdjusted(String sku, int delta, String reason) {}@Servicepublic class StockAdjustmentPublisher { private final KafkaTemplate<String, Object> kafka; public StockAdjustmentPublisher(KafkaTemplate<String, Object> kafka) { this.kafka = kafka; } /** Local transaction: all adjustments of a stocktake become visible together, or none. */ public int publishStocktake(List<StockAdjusted> adjustments) { return kafka.executeInTransaction(ops -> { adjustments.forEach(a -> ops.send("inventory.stock-adjusted", a.sku(), a)); return adjustments.size(); }); } /** The same with declarative transactions. The manager must be named: it is not the primary one. */ @Transactional("kafkaTransactionManager") public void publishStocktakeDeclaratively(List<StockAdjusted> adjustments) { adjustments.forEach(a -> kafka.send("inventory.stock-adjusted", a.sku(), a)); }}Why it is written this way
executeInTransactioncommits when the callback returns and rolls back when it throws. Use theopsargument for the sends. It always starts its own transaction, even if another one is already active on the thread.- Both methods give atomic writes to Kafka only. Neither includes consumer offsets, so neither is consume-process-produce.
- In a service with
transaction-id-prefixset, a barekafka.send(...)in a method with no transaction fails withIllegalStateException. These two shapes are how non-listener code sends. @Transactionalworks through a proxy. CallingpublishStocktakeDeclarativelyfrom another method of the same class bypasses it, and the send then fails for lack of a transaction.
package com.example.inventory;import javax.sql.DataSource;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.context.annotation.Primary;import org.springframework.jdbc.support.JdbcTransactionManager;@Configurationpublic class TransactionManagerConfig { /** * Declared explicitly: once a KafkaTransactionManager exists, do not rely on Spring Boot * to add the database one. Primary, so a plain @Transactional means the database. */ @Bean @Primary JdbcTransactionManager transactionManager(DataSource dataSource) { return new JdbcTransactionManager(dataSource); }}Why it is written this way
- Boot's JDBC and JPA transaction manager beans are conditional on no other
TransactionManagerbean being present.KafkaTransactionManageris aTransactionManager. - With JPA, declare a
JpaTransactionManagerbuilt from theEntityManagerFactoryin the same way. - Boot's
KafkaTransactionManageris still created (its condition only looks for another bean of its own type) and is still the one wired into the listener container.
package com.example.inventory;import java.util.UUID;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;record ReservationRequested(UUID eventId, UUID orderId, String sku, int quantity) {}record StockReserved(UUID eventId, UUID orderId, String sku, int quantity) {}class OutOfStockException extends RuntimeException { OutOfStockException(String sku) { super("No stock left for " + sku); }}@Componentpublic class StockReservationListener { private final JdbcTemplate jdbc; private final KafkaTemplate<String, Object> kafka; public StockReservationListener(JdbcTemplate jdbc, KafkaTemplate<String, Object> kafka) { this.jdbc = jdbc; this.kafka = kafka; } /** * Two transactions. The container opened the Kafka one before this call. * @Transactional opens the database one, which commits when this method returns. * The container commits the Kafka transaction after that. */ @KafkaListener(id = "stock-reservation", topics = "inventory.reservation-requested", groupId = "inventory-service") @Transactional("transactionManager") public void reserve(ReservationRequested request) { // reservations.order_id is UNIQUE: a redelivered request inserts nothing. int inserted = jdbc.update(""" INSERT INTO reservations (order_id, sku, quantity) VALUES (?, ?, ?) ON CONFLICT (order_id) DO NOTHING """, request.orderId(), request.sku(), request.quantity()); if (inserted == 1) { int updated = jdbc.update( "UPDATE stock SET available = available - ? WHERE sku = ? AND available >= ?", request.quantity(), request.sku(), request.quantity()); if (updated == 0) { throw new OutOfStockException(request.sku()); } } // Sent on every run, including a redelivery: the first run's record was aborted // if the Kafka commit failed after the database commit. kafka.send("inventory.stock-reserved", request.orderId().toString(), new StockReserved(request.eventId(), request.orderId(), request.sku(), request.quantity())); }}Why it is written this way
@Transactional("transactionManager")names the database manager. The Kafka transaction is not started by this annotation; the container started it.- Order of events at the end: the method returns, the database commits, then the container sends the offset and commits the Kafka transaction. A failure of the second commit redelivers the request with the reservation already stored.
ON CONFLICT (order_id) DO NOTHING(PostgreSQL) makes the second run a no-op in the database. The stock is decremented only when the reservation row was really inserted.- The send is outside the
if. On a redelivery the database is already correct but the output was aborted, so the event has to be produced again.eventIdcomes from the request, so a consumer that does see it twice can deduplicate. OutOfStockExceptionrolls back both: the database because the method throws, Kafka because the container sees the exception. Mark it not retryable on the after-rollback processor (addNotRetryableExceptions) and route it to a dead-letter topic or, better, publish astock-rejectedevent as a normal result.- An outbox version of this listener writes the
StockReservedevent to an outbox table in the same database transaction and needs no transactional producer. See the transactional outbox.
Configuration
Spring Boot properties first, then the Kafka settings they depend on. Client defaults are those of kafka-clients 3.9 (managed by Spring Boot 3.5); broker defaults are from the Kafka 4.1 reference and are the same in 3.9.
spring.kafka.producer.transaction-id-prefixDefault none (transactions off)
- What it does
- Makes the auto-configured producer factory transactional. Boot then also creates a
KafkaTransactionManagerand sets it on the default listener container factory. - When to change it
- Set it only in services that need atomic writes or consume-process-produce. It must be unique per application instance, and it makes every send require a transaction.
spring.kafka.consumer.isolation-levelDefault read-uncommitted
- What it does
- Maps to the consumer's
isolation.level. Withread-committedthe consumer returns transactional records only after their transaction has committed. - When to change it
- Set
read-committedon every consumer of a transactionally written topic. With the default, consumers also process records of aborted transactions.
transaction.timeout.ms (producer)Default 60000 (1 minute)
- What it does
- How long the coordinator waits for a transaction to finish before it aborts it.
- When to change it
- Lower it so abandoned transactions stop blocking
read_committedreaders sooner, but keep it above your slowest listener call. Set it throughspring.kafka.producer.properties.
transaction.max.timeout.ms (broker)Default 900000 (15 minutes)
- What it does
- The largest
transaction.timeout.msa broker accepts from a producer. - When to change it
- A producer asking for more is rejected when it initialises. Needing more is a sign that slow work sits inside the transaction.
transactional.id.expiration.ms (broker)Default 604800000 (7 days)
- What it does
- How long the coordinator remembers a
transactional.idthat has sent no transaction status updates. - When to change it
- A producer idle for longer is rejected on its next use. For rarely used producers, set
maxAgeonDefaultKafkaProducerFactorybelow this value so that cached producers are refreshed.
transaction.state.log.replication.factor / transaction.state.log.min.isr (broker)Default 3 / 2
- What it does
- Replication of the internal transaction state topic. The topic is created on first use.
- When to change it
- Leave them alone in production. On a single-broker development cluster transactions cannot start until both are set to 1; the Testcontainers Kafka module does that for you.
KafkaTemplate.allowNonTransactionalDefault false
- What it does
- Whether a transactional template may send outside a transaction, using a separate non-transactional producer.
- When to change it
- Prefer a second, non-transactional producer factory and template for sends that do not belong in a transaction. Mixing both through one template hides which sends are atomic.
spring.kafka.listener.typeDefault single
- What it does
- Record listener or batch listener. In a transactional container it also sets the size of a transaction: one record or one poll.
- When to change it
- Use
batchfor high-volume stages to cut the per-record transaction cost. A failure then rolls back and redelivers the whole batch.
ContainerProperties.stopContainerWhenFencedDefault false
- What it does
- Stops the container when its producer is fenced, instead of carrying on.
- When to change it
- The container cannot tell a fencing caused by a rebalance from one caused by a timed-out or expired
transactional.id. Enable it if a silent skip after expiry would be worse than a stopped listener, and alert on the stop.
Hands-on exercise
Trace the two commits, then break them in the lab
Predict what each topic, the consumer group and the database contain after a failure at three different points of a transactional listener, and confirm the Kafka side in the transactions lab.
Steps · 0 of 7 done
Transactional listener with an idempotent database write
Simulation · not a real clusterCrash after processing, before the commit. See which effects Kafka undoes and why the repeated database write is harmless here.
What you should see
Crash inside the method: no row, no visible event, offset unchanged; the redelivery does everything once. Crash between the commits: the row exists, the event is aborted and invisible to read_committed readers, the offset is unchanged; the redelivery inserts nothing, sends the event again, and both commits succeed. Crash after the Kafka commit: row, event and offset are all in place and nothing is repeated. In the lab, the read_uncommitted reader sees the aborted output and the read_committed reader does not.
What can go wrong
Sends that fail after the prefix was added
- Symptom
- A REST endpoint that publishes an event starts returning 500 with
IllegalStateException: No transaction is in processright after transactions were enabled for a listener. - Cause
- The prefix makes the shared
KafkaTemplatetransactional, and a transactional template refuses to send outside a transaction. - Fix
- Wrap such sends in
executeInTransactionor a@Transactional("kafkaTransactionManager")method, or give that code path its own non-transactional producer factory and template.
The same prefix on every instance
- Symptom
- With two or more replicas, listeners log
ProducerFencedExceptionover and over and throughput collapses; one replica alone works. - Cause
- All instances generate the same
transactional.idvalues (prefix plus counter). Each new producer fences the one with the same id on another instance. - Fix
- Include a per-instance value, such as the pod name, in
transaction-id-prefix.
Rollbacks that downstream never notices
- Symptom
- The order service acts on confirmations that the router aborted. The router's logs show
Transaction rolled backfor exactly those records. - Cause
- The order service's consumer uses the default
isolation.level=read_uncommitted, which returns aborted records too. - Fix
- Set
spring.kafka.consumer.isolation-level=read-committedin every service that consumes transactionally written topics.
@Transactional on the wrong manager
- Symptom
- After adding the prefix, JPA or JDBC changes in a
@Transactionallistener are no longer rolled back on an exception, or the application fails to start with more than one transaction manager candidate. - Cause
- Two
PlatformTransactionManagerbeans exist, or the database one is no longer auto-configured, and the annotation does not say which to use. - Fix
- Declare the database transaction manager explicitly as
@Primaryand name the manager in@Transactionalon listeners and Kafka-only methods.
Skipping the send on a duplicate
- Symptom
- A reservation exists in the database but
inventory.stock-reservedwas never published; the order waits for ever. It happened during a broker restart. - Cause
- The database committed, the Kafka commit failed, and on redelivery the listener saw the existing row and returned early without sending. The first send had been aborted.
- Fix
- On a detected duplicate, skip the database change but still produce the output, with the same event id. Or move the output to an outbox row written with the reservation.
Slow work inside the transaction
- Symptom
- Intermittent aborted transactions and fenced producers under load; consumers of the output topic show lag that clears in bursts.
- Cause
- An HTTP call or a slow query inside the listener keeps the Kafka transaction open past
transaction.timeout.ms, and open transactions hold backread_committedreaders. - Fix
- Keep remote calls out of transactional listeners. Do the slow step in a separate, idempotent stage and keep transactions short.
How to troubleshoot
A read_committed consumer of the output topic stops although the producer is writing
Check
kafka-transactions.sh --bootstrap-server localhost:9092 list, then kafka-transactions.sh --bootstrap-server localhost:9092 describe --transactional-id payment-result-router-pod-0-0
- How to read it
- A transaction in state
Ongoingwith an old start time is holding back the last stable offset. Its producer hung or died; it is aborted whentransaction.timeout.msexpires, or when the sametransactional.idinitialises again.
You need to know whether a record was committed or aborted
Check
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.order-confirmed --from-beginning --isolation-level read_committed
- How to read it
- Run it again without the isolation option (the default is read_uncommitted). A record that shows only in the second run belongs to an aborted or still-open transaction.
The application hangs or fails at the first transactional send on a local single broker
Check
Producer log: a TimeoutException while awaiting InitProducerId; broker settings transaction.state.log.replication.factor and transaction.state.log.min.isr
- How to read it
- The internal transaction state topic cannot be created with replication factor 3 on one broker. Set both settings to 1 for local development only.
The same record is processed again and again in a transactional listener
Check
Service log at ERROR: `Transaction rolled back`, followed by the listener's exception; then the after-rollback processor's configuration
- How to read it
- Each rollback seeks back to the record. Without a recoverer it is retried ten times and then only logged. Configure a
DefaultAfterRollbackProcessorwith a dead-letter recoverer.
Offsets of the group do not move even though output is produced
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-result-router
- How to read it
- In a transactional container offsets are committed through the producer as part of the transaction. If
CURRENT-OFFSETstands still while output appears for read_uncommitted readers, the transactions are not committing: look for commit failures or fencing in the log.
Key takeaways
spring.kafka.producer.transaction-id-prefixmakes the producer factory and template transactional, adds aKafkaTransactionManager, and puts every default listener container in a transaction.- Use
executeInTransactionor@Transactional("kafkaTransactionManager")for atomic writes; only the listener container adds the consumed offsets and gives consume-process-produce. - In a transactional container the
AfterRollbackProcessorreplaces the error handler. Configure it with a recoverer, a back-off andcommitRecovered. - A rollback protects only consumers that read with
isolation.level=read_committed. - With a database, Spring commits the database first and Kafka second. A failure in between redelivers the record on the consumer side and is repaired by nothing on the producer side.
- Make the database write idempotent and still send the output on a redelivery, or use an outbox, which is usually the simpler design when a database is involved.
- Give every instance its own prefix, keep transactions short, and declare the database transaction manager explicitly.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A team sets spring.kafka.producer.transaction-id-prefix: router- in a service with 3 replicas. A single replica in the test environment works. In production the logs fill with ProducerFencedException, listeners keep rolling back and lag grows on all 12 partitions.
What should they change first?
Scenario 2 of 3
An inventory listener runs in a container-managed Kafka transaction and is @Transactional on the JDBC manager. It inserts a reservation (unique on order_id), returns early if the row already exists, and otherwise sends stock-reserved. After a broker restart, 6 orders have a reservation row but no stock-reserved event, and their offsets have been committed.
Which change fixes the cause?
Scenario 3 of 3
An order service handles POST /orders: it inserts an order with JPA and must publish OrderPlaced. A developer proposes setting a transaction-id-prefix so the @Transactional method commits the database and then Kafka. Load is 200 orders per second and a lost event means an order that is never fulfilled.
What is the stronger design for this service?
References
References: Spring for Apache Kafka 3.3 reference: Transactions (overview, KafkaTransactionManager, synchronization, local transactions, transactionIdPrefix) ↗ · Spring for Apache Kafka 3.3 reference: Exactly Once Semantics (container-managed transactions, EOSMode.V2) ↗ · Spring for Apache Kafka 3.3 reference: Examples of Kafka Transactions with Other Transaction Managers ↗ · Spring for Apache Kafka 3.3 reference: Handling Exceptions (After Rollback Processor) ↗ · Spring Boot 3.5 reference: Apache Kafka support (transaction-id-prefix, beans attached to the container factory) ↗ · Spring Boot 3.5: Managed dependency coordinates (spring-kafka 3.3.x, kafka-clients 3.9.x) ↗ · Apache Kafka 3.9 producer configuration reference (transaction.timeout.ms, transactional.id) ↗ · Apache Kafka 4.1 broker configuration reference (transaction.max.timeout.ms, transactional.id.expiration.ms, transaction state log settings) ↗ · KIP-447: Producer scalability for exactly once semantics (fencing by consumer group metadata) ↗ · Report a problem