Skip to content

Path 5 · lesson 4 of 5

Advanced22 minApache Kafka 4.x (documentation for 4.1) · Java 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 (Boot 3.5 manages kafka-clients 3.9)

Kafka transactions and your database

A Kafka transaction covers Kafka partitions and consumer offsets, never your database. See the dual-write failure windows and the designs that close them.

What you will learn

  • Name the two failure windows of a dual write and say what the database and the topic contain after a crash in each
  • Explain why a Kafka transaction and a JDBC transaction cannot be committed as one atomic unit
  • Describe exactly what Spring's transaction synchronisation does, in which order it commits, and which window remains
  • Choose between a transactional outbox, change data capture, offsets stored in the database and an idempotent consumer for a given service
  • Read a payment or order incident and decide which side of the boundary lost or duplicated the effect

Before you start

The concept

An order service does two things when a customer checks out: it inserts a row into its orders table, and it publishes OrderPlaced so that payment, inventory and notifications can react. Those are writes to two different systems. The database commits when the database says so. Kafka accepts the record when the partition leader says so. Nothing ties the two decisions together, so the service can die after one and before the other. This is the dual-write problem.

It is tempting to reach for Kafka transactions here, because the word is the same. A Kafka transaction makes a set of writes to Kafka partitions, plus the consumer offsets sent through the producer, commit or abort together. That is the whole of its scope. A row in PostgreSQL, a charge at a payment provider and an email are outside it, and no client setting moves them inside.

What you want to be atomicCovered by a Kafka transaction?What covers it instead
Records written to several topic partitionsYesThe transactional producer
Those records plus the offsets of the input you consumedYessendOffsetsToTransaction (Spring's listener container calls it for you)
An INSERT into orders plus the OrderPlaced recordNoOutbox or CDC: one database transaction, publishing derived from it
A consumed record's offset plus an UPDATE in your databaseNoIdempotent write, or the offset stored in the same database transaction
A card charge at a provider plus anything elseNoAn idempotency key the provider honours, and a retry
The boundary of a Kafka transaction is Kafka.

Why it exists

Take the two possible orders of a dual write. In both, the code looks correct and passes every test, because tests rarely kill the process between two lines.

Order of the two writesCrash or failure hereDatabaseKafkaWhat the business sees
Database commit, then sendAfter the commit, before the send is acknowledgedOrder 7001 existsNo OrderPlaced for 7001A paid-for basket that is never charged, never reserved, never shipped. Nothing retries it, because the request already returned or died.
Database commit, then sendThe send fails after the commit (broker unavailable, delivery.timeout.ms expires)Order 7001 existsNo OrderPlaced for 7001Same as above, and the caller may also receive an error for an order that was in fact saved.
Send, then database commitAfter the send is acknowledged, before the commitNo order 7002OrderPlaced for 7002 is in the logThe payment service charges a card for an order that does not exist. Support cannot find it.
Send, then database commitThe commit fails (constraint violation, deadlock victim, lost connection)No order 7002OrderPlaced for 7002 is in the logA phantom event. You cannot take a record back out of a partition.
Either orderThe send times out but the broker did append the recordDepends on what the code does with the exceptionRecord is thereCode that rolls the order back on a send error has now produced a phantom event from a different direction.
Crash here, and this is what the world looks like. Neither order is safe.

Before patterns such as the outbox became routine, teams lived with these windows and ran reconciliation jobs to find orders without events. The windows are narrow, so the bug shows up as a handful of stuck orders per month, usually during a deploy or a broker restart, which is exactly when many processes are being stopped between two lines.

Outbox pattern

Simulation, not a real cluster

The animated Build lesson shows the same dual write failing and the outbox closing the window. Simulated, not a real broker.

Step 1 · The pattern

Save it, then tell everyone

An order comes in. The service saves it to its database, then publishes an event so shipping, email and analytics can react. Two writes, to two systems.

  • order request
  • done
Open the full animated lesson

How it works inside

Why Kafka cannot enlist your database

Inside Kafka, a transaction is run by a transaction coordinator, a broker that records each transaction's state in an internal log. The producer registers the partitions it writes to, and on commit the coordinator first records the decision durably and then writes commit markers into every partition involved, including the consumer offsets partition when offsets were sent. That is a two-phase protocol, but every participant is a Kafka partition. KIP-98, which introduced it, defines no step in which an outside resource can be asked to prepare and then told the outcome.

A relational database has its own commit protocol and its own log. To commit both atomically you would need a coordinator that both sides obey: each side would first promise it can commit (prepare), and only then be told to commit. The standard Kafka producer API offers beginTransaction, commitTransaction and abortTransaction, with no prepare step you can hand to a JTA or XA transaction manager. Spring therefore has two separate transaction managers for the two systems, and the best it can do is call them one after the other.

What Spring's transaction synchronisation gives you

With a transactional producer factory (a transaction-id-prefix is set) and a method annotated @Transactional that is bound to the database transaction manager, a KafkaTemplate.send inside the method starts a Kafka transaction and registers it with the database transaction. When the method returns, Spring commits the database transaction and then commits the Kafka transaction. If the method throws, both roll back. The Spring reference describes this as synchronisation; it is two commits in a fixed order, not one.

SetupCommit orderWindow that remainsResult of a failure in the window
Producer side, @Transactional on the database manager, template synchronisesDatabase, then KafkaDatabase committed, Kafka commit fails or the process diesOrder row without an event. Current versions throw the failure to the caller, who must compensate; a dead process cannot.
Producer side, nested @Transactional: outer database manager, inner KafkaTransactionManagerKafka, then databaseKafka committed, database commit failsEvent without an order row.
Consumer side: the listener container starts the Kafka transaction, the listener method is @Transactional on the database managerDatabase, then offsets and KafkaDatabase committed, Kafka commit fails or the process diesOutput records are aborted and the input record is delivered again, so the database write runs a second time.
Ordered, best-effort commits. Each row still has a window.

The third row is the useful one. A consumer has something the producer side lacks: the input record is still in the log, and its offset did not move, so the work is retried automatically. The Spring reference says as much: when the Kafka commit fails after the database commit, the record is redelivered, so the database update must be idempotent. Its chapter on exactly-once semantics adds that the read and the process steps keep at-least-once semantics; only the read-process-write sequence as a whole completes once in Kafka.

Where the Kafka transaction ends

Simulation · not a real cluster

Transactional mode: offsets and output records commit together. Crash after processing and watch which effects are rolled back and which one is not.

The designs that actually close the window

DesignHow it removes the dual writeWhat you still have to handle
Transactional outboxThe event is a row inserted in the same database transaction as the business row. A relay publishes rows afterwards.The relay publishes at least once, so consumers must tolerate duplicates. Ordering per key needs care in the relay.
Change data captureA connector reads the database's own commit log and publishes what was committed, often from an outbox table.Operating Kafka Connect and the connector. Still at-least-once towards consumers after a connector restart.
Offsets stored with the dataThe consumer writes its result and the next offset to read in one database transaction, and seeks to the stored offset when a partition is assigned.You own offset storage and seeking (ConsumerSeekAware in Spring). Kafka's committed offsets and lag tooling no longer describe the truth for that group.
Idempotent consumerThe write is safe to repeat: a unique key or a processed-events row in the same database transaction rejects the second attempt.Every event needs a stable identity, and the dedupe record must be kept long enough.

The Kafka design documentation points at the third design itself: when writing to an external system, let the consumer keep its position in the same place as its output, because many stores cannot take part in a two-phase commit. All four designs replace two independent commits by one commit in one system plus something that can be repeated safely.

Architecture

Drawing diagram…

A payment service writes to two boxes. One box, owned by Kafka's transaction coordinator, holds output records and consumer offsets. The other, owned by the database engine, holds the payments, outbox and processed-events rows. A card provider call sits outside both.

flowchart LR
  IN["orders.order-placed"] --> APP["Payment Service"]
  subgraph KTX["Kafka transaction"]
    OUT["payments.payment-result records"]
    OFF["Offsets of group payment-service"]
  end
  subgraph DBTX["Database transaction"]
    PAY["payments row"]
    OBX["outbox row"]
    DED["processed_events row"]
  end
  APP --> OUT
  APP --> OFF
  APP --> PAY
  APP --> OBX
  APP --> DED
  TC["Transaction coordinator on a broker"] --> OUT
  TC --> OFF
  DBE["Database engine and its log"] --> PAY
  APP -.-> EXT["Card provider HTTP call, in neither transaction"]
Two transaction boundaries with two different owners, and an HTTP call that belongs to neither.
  • The left box commits when the transaction coordinator writes commit markers. The right box commits when the database flushes its own log. Neither waits for the other.
  • Everything inside one box is all-or-nothing. That is why the outbox row and the processed-events row live in the database box next to the business row: they borrow the database's atomicity.
  • The provider call is in neither box. It can only be made safe by being repeatable, which the next lesson covers.
  • A design is sound when a crash at any point leaves each box internally consistent and the next retry brings the other box up to date.

Message flow

Drawing diagram…

Sequence diagram with an order service, a database and Kafka. For order 7001 the database commits, the service crashes, and the event is never sent. For order 7002 the event is sent and acknowledged, the service crashes, and the database rolls back.

sequenceDiagram
  participant S as Order Service
  participant D as Orders DB
  participant K as Kafka
  Note over S,K: Order 7001, database first
  S->>D: BEGIN and INSERT order 7001
  S->>D: COMMIT
  D-->>S: committed
  Note over S: crash here
  S-xK: send OrderPlaced 7001 never happens
  Note over D,K: 7001 is saved but no service hears about it
  Note over S,K: Order 7002, Kafka first
  S->>D: BEGIN and INSERT order 7002
  S->>K: send OrderPlaced 7002
  K-->>S: acknowledged
  Note over S: crash here
  S-xD: COMMIT never happens
  D-->>D: connection lost, rollback
  Note over D,K: 7002 is in the topic but not in the database
The same crash, in the two possible orders of a dual write.
  1. Order 7001: the commit is durable, so restarting the service does not undo it. No code path remembers that an event is owed.
  2. Order 7002: the record is in the partition and consumers will read it. The database rolled back because the connection dropped before the commit.
  3. Wrapping the send in a Kafka transaction changes only the second half: an uncommitted record can be aborted. It moves the window to between the two commits; it does not remove it.
  4. With an outbox, the first sequence becomes safe: the owed event is itself a committed row, so a relay finds and publishes it after any restart.

In a real system

A typical checkout: the order service owns a PostgreSQL schema and publishes to orders.order-placed, 12 partitions, keyed by orderId so that every event about one order stays in one partition. The payment service consumes it in group payment-service, charges the card and publishes payments.payment-result, also keyed by orderId.

  • Order service (producer side). It has no input record to be redelivered, so nothing repairs a lost send. It writes the order and an outbox row in one database transaction. A relay or a CDC connector publishes the row. See the lab Transactional outbox.
  • Payment service (consumer side). It is redelivered automatically after a crash, so it needs repeatable writes: a unique constraint on order_id in payments, and its own outbox row for payments.payment-result in the same database transaction as the status change.
  • Ledger projection (consumer only, no outgoing events). A consumer that builds a balance table can store (topic, partition, next_offset) in the same transaction as each balance update and seek to it on assignment. A replayed record then lands on an offset the database has already passed and is skipped.
  • Fraud scoring (Kafka in, Kafka out, no database). This is the one place where a Kafka transaction alone is enough: read payments.payment-result, write fraud.scores, commit offsets and output together.

Java and Spring Boot

The order service, first as the dual write to recognise in code review, then with an outbox. PostgreSQL syntax. The relay shown is the smallest correct one; the transactional outbox lesson covers ordering, batching and CDC.

order-service/src/main/java/com/example/orders/DualWriteOrderService.javaJava
package com.example.orders;import java.time.Instant;import java.util.UUID;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.transaction.annotation.Transactional;/** Do not copy: this is the dual write. Kept here so you can recognise it. */public class DualWriteOrderService {    private final JdbcTemplate jdbc;    private final KafkaTemplate<String, OrderPlaced> kafka;    public DualWriteOrderService(JdbcTemplate jdbc, KafkaTemplate<String, OrderPlaced> kafka) {        this.jdbc = jdbc;        this.kafka = kafka;    }    @Transactional    public UUID placeOrder(PlaceOrder cmd) {        UUID orderId = UUID.randomUUID();        jdbc.update(            "INSERT INTO orders (order_id, customer_id, amount_minor, currency, status) VALUES (?, ?, ?, ?, 'PLACED')",            orderId, cmd.customerId(), cmd.amountMinor(), cmd.currency());        // The send leaves this method before the database commit below has happened.        kafka.send("orders.order-placed", orderId.toString(),            new OrderPlaced(UUID.randomUUID(), orderId, cmd.customerId(),                cmd.amountMinor(), cmd.currency(), Instant.now()));        return orderId;        // The database commits here, after the method returns.        // Commit fails -> the event is already on its way. Send fails later -> the order is already saved.    }}

Why it is written this way

  • send is asynchronous: it returns a future once the record is in the producer's buffer. The method can return, and the database can commit, before the broker has accepted or rejected the record.
  • The @Transactional here is bound to the database only. Without a transaction-id-prefix the template is not transactional and takes no part in it at all.
  • Setting a transaction-id-prefix would make Spring commit the database first and Kafka second. The window between those two commits remains, and on this producer side nothing retries the request.
  • Waiting on the future inside the method does not help either: it only chooses which of the two failure windows you get, and it holds a database connection during a network call.

Configuration

Settings that decide where the boundary sits. Kafka defaults are from the 4.1 configuration reference; Spring behaviour is from the Spring for Apache Kafka 3.3 and Spring Boot 3.5 references.

spring.kafka.producer.transaction-id-prefixDefault none
What it does
Makes the producer factory transactional. Spring Boot then also configures a KafkaTransactionManager and wires it into the listener container. The prefix must differ on each application instance.
When to change it
Set it for read-process-write between topics. Setting it does not bring a database into the transaction; it only lets Spring order the two commits.
isolation.level (consumer)Default read_uncommitted
What it does
With read_committed, poll returns transactional records only once their transaction has committed, and nothing beyond the first still-open transaction.
When to change it
Set read_committed on every consumer of a topic written transactionally. With the default, consumers also receive records from aborted transactions, which defeats the rollback.
transaction.timeout.ms (producer)Default 60000 (1 minute)
What it does
How long a transaction may stay open before the coordinator aborts it.
When to change it
A slow database call inside a synchronised transaction counts against it. Keep database work short rather than raising this; a long open transaction also holds back read_committed consumers.
enable.auto.commit (consumer)Default true (Spring for Apache Kafka sets it to false unless you set it explicitly)
What it does
Whether the consumer commits offsets on a timer, independently of whether processing finished.
When to change it
Leave it off, as Spring does, so that offsets advance only after your database write has committed. Timed commits can move the offset past a record whose write later fails.
transaction.two.phase.commit.enable (producer)Default false (listed from the 4.1 reference; absent in 4.0)
What it does
From KIP-939: tells the broker that this producer takes part in a two-phase commit driven by an external coordinator, so its transactions do not time out.
When to change it
Leave it alone unless a framework that implements such a coordinator tells you to set it. It does not make Spring's @Transactional atomic across Kafka and JDBC.

Hands-on exercise

Find the edge of the transaction

Use the delivery simulator in transactional mode to see which effects a Kafka transaction rolls back after a crash, and which effect it cannot reach.

Steps · 0 of 6 done

Transactional mode with a crash before commit

Simulation · not a real cluster

The Kafka transaction aborts cleanly. Watch what that does and does not undo for the payment.

Take it further in the lab: Database unavailable while consuming Kafka messages

What can go wrong

Trusting @Transactional to cover the send
Symptom
A few orders per month are in the database with status PLACED and no downstream service has ever heard of them. They cluster around deploys.
Cause
The method's transaction is the database's. The Kafka send is a second, independent write; the process stopped, or the send failed, after the commit.
Fix
Write the event to an outbox table in the same database transaction and publish from there. Add a query that alerts on old unpublished rows.
Phantom events from a rollback
Symptom
The payment service logs charges for order ids that the order service cannot find.
Cause
The event was sent inside the method, and the database then rolled back (constraint violation, deadlock, lost connection). The record stays in the partition.
Fix
Never send from inside an open database transaction. Use the outbox, so that an event can only exist if its row committed.
Reading uncommitted records downstream
Symptom
A consumer acts on events that the producing service aborted, even though that service uses Kafka transactions correctly.
Cause
The consumer runs with the default isolation.level=read_uncommitted, which returns aborted transactional records as well.
Fix
Set isolation.level=read_committed on every consumer of a transactionally written topic.
A non-idempotent write under a synchronised listener
Symptom
A ledger shows the same payment credited twice after a broker restart, although the service uses exactly-once settings.
Cause
The database committed, the Kafka commit that follows it failed, and the record was redelivered. The second run inserted a second row.
Fix
Make the database write idempotent: a unique constraint on the business key, or a processed-events row in the same database transaction.
Holding the Kafka transaction open across slow work
Symptom
Intermittent aborted transactions and producer errors during database slowness; read_committed consumers of the output topic stall.
Cause
Slow database or HTTP work inside the transaction exceeds transaction.timeout.ms, so the coordinator aborts it.
Fix
Keep remote calls out of the transaction, shorten the database work, and prefer an outbox over a synchronised Kafka transaction for database-backed services.

How to troubleshoot

Orders exist but no event was published

Check

SELECT o.order_id FROM orders o LEFT JOIN outbox x ON x.record_key = o.order_id::text WHERE x.id IS NULL AND o.created_at < now() - interval '5 minutes';
How to read it
Rows returned were written without an outbox row: some code path still does a dual write. With an outbox in place this query must always be empty.
Events are delayed by minutes

Check

SELECT count(*), min(created_at) FROM outbox WHERE published_at IS NULL;
How to read it
A growing count with an old minimum means the relay is stopped or failing to publish. Look for its send errors; orders are safe but downstream services are waiting.
A consumer seems stuck although the producer is writing

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
Lag that grows on one partition while CURRENT-OFFSET stands still can be a read_committed consumer waiting behind an open transaction. Find the producer that started a transaction and never finished it.
Downstream services saw an event the producer says it rolled back

Check

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic payments.payment-result --from-beginning --isolation-level read_committed
How to read it
If the record is absent here but your service received it, the service is reading uncommitted. If it is present, the Kafka transaction did commit and the rollback was on the database side only.

Key takeaways

  • A database commit and a Kafka send are two commits in two systems. Whatever order you pick, a crash between them leaves one side without the other.
  • A Kafka transaction is atomic over Kafka partitions and consumer offsets, and over nothing else.
  • Spring's transaction synchronisation commits one transaction and then the other. It narrows the window and chooses its direction; it does not close it.
  • On the producer side use an outbox or CDC: one database transaction, with publishing derived from committed rows.
  • On the consumer side rely on redelivery and make the database write idempotent, or store the offset in the same database transaction as the data.
  • Every one of these designs delivers at least once somewhere, so downstream consumers still need to deduplicate.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

An order service runs @Transactional placeOrder(): it inserts the order, then calls kafkaTemplate.send. No transaction-id-prefix is set. After a rolling deploy, support reports 3 orders from that minute with status PLACED and no payment attempt. The topic has no events for them and the broker logs show no errors.

What should the team change?

Scenario 2 of 3

A payment listener runs in a container-managed Kafka transaction and its method is @Transactional on the JDBC manager. It inserts into ledger_entries (no unique constraint) and sends payment-result. During a broker restart, 14 ledger entries appear twice, while payment-result contains each result exactly once.

Which explanation fits both observations?

Scenario 3 of 3

A consumer builds an account_balance table from ledger.entries (6 partitions). Each event adds an amount to a balance, so applying one twice is wrong, and the events carry no unique id. The service writes to no other topic. The team wants each event applied exactly once in the table.

Which design achieves that with the least machinery?

References

References: Apache Kafka documentation: Design, message delivery semantics and using transactions ↗ · KIP-98: Exactly Once Delivery and Transactional Messaging ↗ · KafkaProducer Javadoc (4.1): idempotent and transactional producer ↗ · Spring for Apache Kafka 3.3 reference: Transactions ↗ · Spring for Apache Kafka 3.3 reference: Exactly Once Semantics ↗ · Spring for Apache Kafka 3.3 reference: Examples of Kafka transactions with other transaction managers ↗ · Apache Kafka 4.1 producer configuration reference ↗ · KIP-939: Support Participation in 2PC ↗ · Report a problem