Skip to content

Path 7 · lesson 1 of 8

Intermediate30 minApache Kafka 4.x (configuration reference read for 4.1) · Spring for Apache Kafka 3.3 on Spring Boot 3.x · Java 21 · PostgreSQL

Transactional outbox

Write the event to a table in the same database transaction as the business row, then let a relay publish it to Kafka. No event is lost, and none is published for a rolled-back change.

What you will learn

  • Explain why writing to a database and to Kafka from one method cannot be made atomic, and what each failure order leaves behind
  • Design an outbox table and write the event row in the same local transaction as the business row
  • Implement a polling relay in Spring Boot that publishes outbox rows in order and marks them only after the broker acknowledges
  • State what the pattern guarantees (at-least-once publication) and build the consumer-side deduplication that it requires
  • Keep events for one aggregate in order, and keep the outbox table small

Before you start

The concept

The Order Service does two things when a customer checks out: it stores the order in its own database, and it tells the rest of the system by publishing OrderCreated to Kafka. Payment, Inventory and Notification all start from that event. The two writes go to two different systems, and there is no transaction that covers both. This is the dual-write problem.

The transactional outbox removes the second write from the request path. Instead of calling Kafka, the service inserts a row describing the event into an outbox_event table in the same database transaction as the order row. Either both rows commit or neither does. A separate process, the message relay, later reads the outbox and publishes each row to Kafka.

  • Outbox table: one row per event, in the service's own database: an event id, the aggregate it belongs to, the event type and the payload.
  • Business transaction: INSERT INTO orders and INSERT INTO outbox_event, one commit.
  • Relay: publishes rows to Kafka after they commit. It is either a polling publisher (a loop that queries for unpublished rows, built in this lesson) or log-based change data capture (a tool that reads the database's transaction log, covered in the next lesson).
  • Idempotent consumers: the relay can publish a row twice, so every consumer must recognise an event it has already handled.

Advantages

  • The business change and its event are atomic, using only a local database transaction
  • Placing an order does not depend on Kafka being reachable: events queue in the table during a broker outage
  • No two-phase commit and no Kafka transaction spanning a database
  • Events for one aggregate are published in the order they were committed
  • The outbox table is an inspectable record of what was and was not sent

Disadvantages

  • Publication is at-least-once, so every consumer needs deduplication
  • Events arrive after a delay: the poll interval, or the CDC lag
  • Every business transaction pays for one more insert
  • A relay is one more thing to run, monitor and alert on
  • The table grows without cleanup, and a stuck relay silently stops all downstream work
  • Developers must remember to write the outbox row: nothing forces it

Outbox pattern

Simulation, not a real cluster

The animated lesson shows the same dual write failing and the outbox closing the gap, step by step.

Step 1 · The pattern

Save it, then tell everyone

An order comes in. The service saves it to its database, then publishes an event so shipping, email and analytics can react. Two writes, to two systems.

  • order request
  • done
Open the full animated lesson

Why it exists

Take the obvious implementation: orderRepository.save(order) followed by kafkaTemplate.send(...). Whichever order you choose, there is a moment between the two calls where a crash, a timeout or a rollback leaves the two systems disagreeing.

Order of the two writesWhat goes wrongWhat the business sees
Commit to the database, then send to KafkaThe process dies, or the send fails, after the commitAn order that is never paid, never reserved and never confirmed. Nothing retries it, because nothing knows the event is missing.
Send to Kafka, then commit to the databaseThe commit fails (constraint violation, deadlock, lost connection)A ghost event: Payment charges the customer for an order id that does not exist.
Send inside the database transaction, before the commitThe same as the row above: a Kafka send cannot be rolled back by the databaseA ghost event again. send is also asynchronous, so the commit can succeed while the send later fails.
Kafka transaction around bothA Kafka transaction covers writes to Kafka partitions and consumer offset commits, and nothing elseTwo separate commits that can still disagree. Spring can synchronise them, but that only shrinks the window.

Spring for Apache Kafka can synchronise a Kafka transaction with a database transaction manager. The reference is explicit about what that means: the database transaction commits first and the Kafka transaction after it, and if the second commit fails the exception is thrown to the caller, who must take remedial action. It is a best-effort ordering of two commits. A crash between them still loses the event.

A distributed transaction (XA, two-phase commit) would be the textbook answer, but Kafka does not take part in XA transactions, and two-phase commit ties the availability of checkout to the availability of every participant. The outbox takes a different route: use the one system that already offers atomicity, the service's own database, for both facts. Publishing then becomes a separate problem: copying committed rows to Kafka with retries. That problem is easy to make reliable.

How it works inside

The write side

The outbox row is ordinary data. It is inserted with the same connection and in the same transaction as the business row, so the database's commit makes both durable together, and a rollback removes both. The row carries everything the relay needs to build a Kafka record: a unique event id (so consumers can deduplicate), the aggregate type (which decides the topic), the aggregate id (which becomes the record key) and the serialised payload.

The relay: a polling publisher

  1. Select a batch of rows where published_at IS NULL, ordered by an insertion sequence.
  2. For each row, send a record to Kafka with key = aggregateid and wait for the broker's acknowledgement.
  3. Only after the acknowledgement, set published_at on that row.
  4. Commit, sleep for the poll interval, repeat.

Steps 2 and 3 are themselves a dual write: Kafka first, then the database. The relay cannot make them atomic either. What it can do is choose the safe order. Marking a row before sending could lose the event. Sending before marking can only repeat it. That choice is where at-least-once comes from.

Crash pointResulting stateRecovery
Order Service, before its commitNo order row, no outbox rowNothing to recover. The client got an error and can retry the request.
Order Service, after its commitOrder row and outbox row both exist, unpublishedThe relay finds the row on its next poll and publishes it. No event is lost.
Relay, after reading the batch, before sendingRows still unpublishedThe next poll reads the same rows. No duplicate.
Relay, after Kafka acknowledged, before marking the rowRecord is in Kafka, row still says unpublishedThe next poll sends it again: a duplicate with the same event id, which consumers skip.
Kafka unavailableSends fail or time out, rows stay unpublished and accumulateOrders are still accepted. When the brokers return, the relay drains the backlog in order.
Database unavailableNo orders can be placed and the relay cannot readNothing is half-done. Both resume when the database returns.
Consumer, after its own commit, before the offset commitEffect applied, offset not advancedThe record is redelivered, and the consumer's processed-event check turns it into a no-op.
Crash at each point in the flow, what is left behind, and how the system recovers.

Ordering per aggregate

Kafka orders records only within a partition, so the relay uses the aggregate id (orderId) as the key: all events of one order land in one partition. That is necessary but not sufficient. The relay must also hand them to the producer in commit order, and three things break that:

  • Several relay instances reading at once. With FOR UPDATE SKIP LOCKED, instance B skips the rows instance A has locked and may publish a later event of the same order first. The code below allows only one active relay at a time, using a PostgreSQL advisory lock.
  • Continuing past a failed send. If row 3 fails and the relay goes on to row 4 of the same order, the retry of row 3 arrives after it. The relay stops the batch at the first failure.
  • A cursor on the sequence (WHERE seq > :lastSeen). Sequence values are assigned at insert time, and transactions commit in a different order, so a row with a lower seq can become visible after the cursor has passed it. That row is then never published. Selecting on published_at IS NULL cannot skip a row.

Two transactions that change the same order are serialised by the row lock on that order, so their outbox rows get increasing seq values in commit order. Events of different orders may be published in a slightly different order than they committed, which no consumer should depend on anyway.

Cleanup

Published rows have no further use to the relay. Delete them in small batches on a schedule, keeping a few days for debugging. A partial index on unpublished rows keeps the relay's query fast regardless of how many published rows remain. With a CDC relay the row can be deleted in the same transaction that inserts it, because the relay reads the transaction log and not the table.

Architecture

Drawing diagram…

Flowchart: the Order Service inserts into the orders table and the outbox table in one database transaction. An outbox relay polls unpublished rows, sends them to the orders.events topic keyed by order id, then marks them published. Payment, Inventory and Notification consume the topic, and Payment records processed event ids.

flowchart LR
  Client["Checkout request"] --> OS["Order Service"]
  subgraph DB["Order database, one local transaction"]
    OT["orders table"]
    OB["outbox_event table"]
  end
  OS -->|"INSERT order"| OT
  OS -->|"INSERT event row"| OB
  OB -->|"poll unpublished rows"| RL["Outbox relay"]
  RL -->|"send key orderId, wait for ack"| T["Topic orders.events"]
  RL -->|"mark published_at"| OB
  T --> PS["Payment Service"]
  T --> IS["Inventory Service"]
  T --> NS["Notification Service"]
  PS --> PE["processed_event table"]
The request touches only the database. The relay is the only component that writes order events to Kafka.
  • The Order Service has no Kafka producer on the request path. Its only dependency for accepting an order is its own database.
  • The relay is the single writer of orders.events. One writer per aggregate type, sending in sequence order, is what keeps events of one order in order.
  • The arrow from the relay back to the outbox (mark published_at) happens after the broker's acknowledgement. A crash between the two is the source of duplicates.
  • Each consumer keeps its own processed_event table in its own database, written in the same transaction as its business effect.

Message flow

Drawing diagram…

Sequence diagram: the Order Service inserts the order and the outbox row and commits. The relay reads the row, sends it to Kafka and is acknowledged, then crashes before marking it. After restart it reads and sends the same row again and marks it. The Payment Service receives two copies with the same event id, processes the first and skips the second.

sequenceDiagram
  participant C as Client
  participant O as Order Service
  participant D as Order DB
  participant R as Outbox relay
  participant K as Kafka orders.events
  participant P as Payment Service
  C->>O: POST /orders
  O->>D: BEGIN
  O->>D: INSERT orders row
  O->>D: INSERT outbox_event row
  O->>D: COMMIT
  O-->>C: 201 Created, order PENDING
  R->>D: SELECT unpublished rows ORDER BY seq
  D-->>R: OrderCreated, event id e1
  R->>K: send key o-1001, header id e1
  K-->>R: ack
  Note over R: crash here, before marking the row
  R->>D: SELECT unpublished rows after restart
  D-->>R: OrderCreated e1 again
  R->>K: send key o-1001, header id e1
  K-->>R: ack
  R->>D: UPDATE published_at for e1
  K->>P: OrderCreated e1, first copy
  P->>P: insert e1 into processed_event, create payment
  K->>P: OrderCreated e1, second copy
  P->>P: e1 already processed, skip
A relay crash between the acknowledgement and the mark produces a second copy, which the consumer skips by event id.
  1. The client gets its response as soon as the local transaction commits. Publication happens afterwards, so the API is asynchronous from this point: the order is PENDING until later events arrive.
  2. The duplicate is not a producer retry. It is a second, independent send of the same business event, which is why producer idempotence does not remove it.
  3. Both copies carry the same event id in a header because the id is stored in the outbox row, not generated at send time.

In a real system

A typical order system uses one outbox table per service database. The Order Service writes OrderCreated, OrderConfirmed and OrderCancelled rows with aggregatetype = 'orders'. Its relay publishes them to orders.events, a topic with 12 partitions and replication factor 3, with the order id as key. The Payment Service has its own outbox for payments.events, and Inventory has one for inventory.events. Consumer groups are payment-service, inventory-service and notification-service, each reading orders.events independently.

DecisionChoice in this systemReason
Record keyaggregateid (order id)All events of an order share a partition, so consumers see them in order
Event idUUID stored in the outbox row, sent as header idStable across republication, so consumers can deduplicate
Event typeColumn type, sent as header eventTypeConsumers can route without parsing the payload
PayloadA complete, self-describing JSON eventConsumers must not need to call back to the Order Service to understand it
RelayPolling publisher first, CDC when latency or load requires itPolling needs no extra infrastructure and works on any database
Retention of published rowsDeleted after 3 daysEnough to investigate an incident without letting the table grow

When to use it

  • A service changes its own database and must publish an event about that change, and a missing or phantom event has a business cost (orders, payments, stock, account changes).
  • The service must keep accepting writes while Kafka is unavailable.
  • Consumers can be made idempotent, which they need to be for at-least-once consumption in any case.
  • A delay of a few hundred milliseconds between commit and publication is acceptable.

When not to use it

  • There is no database write to stay consistent with: a service that only transforms records from one topic to another should use Kafka transactions for read-process-write.
  • The data is high-volume telemetry such as clickstream, where an occasional lost record is acceptable and an extra insert per event is not. Produce directly.
  • The consumer needs the row state, not a business event (cache or search index updates). Plain change data capture on the table is simpler than writing outbox rows.
  • The database cannot offer a multi-row transaction across the business table and the outbox table. The pattern depends entirely on that.

Java and Spring Boot

The Order Service on Spring Boot 3 with JdbcTemplate and PostgreSQL: the schema, the transactional write, a polling relay, and a consumer that deduplicates. Serializers are StringSerializer and StringDeserializer: the payload is stored as JSON in the outbox and sent unchanged.

src/main/resources/db/migration/V1__orders_and_outbox.sqlSQL
-- Order Service databaseCREATE TABLE orders (    id           varchar(36) PRIMARY KEY,    customer_id  varchar(36) NOT NULL,    sku          varchar(64) NOT NULL,    quantity     integer     NOT NULL,    total_cents  bigint      NOT NULL,    status       varchar(16) NOT NULL,    created_at   timestamptz NOT NULL DEFAULT now());CREATE TABLE outbox_event (    seq           bigserial    PRIMARY KEY,           -- publication order    id            uuid         NOT NULL UNIQUE,       -- event id, used by consumers to deduplicate    aggregatetype varchar(255) NOT NULL,              -- decides the topic: 'orders' -> orders.events    aggregateid   varchar(255) NOT NULL,              -- becomes the Kafka record key    type          varchar(255) NOT NULL,              -- 'OrderCreated', 'OrderCancelled', ...    payload       jsonb        NOT NULL,    created_at    timestamptz  NOT NULL DEFAULT now(),    published_at  timestamptz                         -- NULL until the broker has acknowledged);-- The relay only ever looks at unpublished rows: keep that lookup small.CREATE INDEX outbox_unpublished ON outbox_event (seq) WHERE published_at IS NULL;-- Transaction 1 (request thread): the business row and the event row, one commit.--   BEGIN;--   INSERT INTO orders (...) VALUES (...);--   INSERT INTO outbox_event (id, aggregatetype, aggregateid, type, payload) VALUES (...);--   COMMIT;-- Transaction 2 (relay): read in order, send to Kafka, mark what was acknowledged.--   BEGIN;--   SELECT pg_try_advisory_xact_lock(7301);--   SELECT ... FROM outbox_event WHERE published_at IS NULL ORDER BY seq LIMIT 100;--   UPDATE outbox_event SET published_at = now() WHERE seq = ?;   -- once per acknowledged row--   COMMIT;-- Cleanup job (scheduled, small batches):--   DELETE FROM outbox_event--   WHERE seq IN (SELECT seq FROM outbox_event--                 WHERE published_at < now() - interval '3 days' LIMIT 5000);-- Payment Service database (a different database): the consumer's dedupe table.CREATE TABLE processed_event (    consumer     varchar(64) NOT NULL,    event_id     uuid        NOT NULL,    processed_at timestamptz NOT NULL DEFAULT now(),    PRIMARY KEY (consumer, event_id));CREATE TABLE payment (    order_id     varchar(36) PRIMARY KEY,             -- one payment per order    amount_cents bigint      NOT NULL,    status       varchar(16) NOT NULL);

Why it is written this way

  • The column names id, aggregatetype, aggregateid, type and payload match the defaults of Debezium's outbox event router, so the same table works when the polling relay is later replaced by CDC.
  • seq gives the relay a publication order. id is a separate UUID because consumers need an identifier that means the same thing in every service.
  • The partial index covers only unpublished rows, so the relay's query cost does not depend on how many published rows are waiting for cleanup.
  • processed_event lives in the consumer's database and is written in the same transaction as the consumer's effect. A dedupe store outside that transaction reintroduces a dual write.

Configuration

The relay's producer should favour durability and order over latency. Kafka defaults below are from the Kafka 4.1 producer configuration reference. The last three are application settings of the relay shown above, not Kafka settings.

acksDefault all
What it does
The leader waits for all in-sync replicas before acknowledging. With min.insync.replicas=2 and replication factor 3 on the topic, an acknowledged event survives one broker failure.
When to change it
Leave at all. The relay marks a row as published on the strength of this acknowledgement, so a weaker setting turns the outbox back into a way to lose events.
enable.idempotenceDefault true
What it does
The broker discards duplicates caused by the producer retrying a batch within its session, and keeps per-partition order across those retries.
When to change it
Leave on. It does not deduplicate a row the relay sends twice after a crash. That is the consumer's job.
max.in.flight.requests.per.connectionDefault 5
What it does
Unacknowledged requests per connection. With idempotence enabled, order is preserved for any value up to 5.
When to change it
No need to lower it while idempotence is on. If idempotence is ever disabled, values above 1 allow reordering on retry.
delivery.timeout.msDefault 120000 (2 minutes)
What it does
Upper bound on the time from send() returning to the producer reporting success or failure, including retries.
When to change it
Consider lowering it towards the relay's own wait so that a send the relay has given up on does not still succeed a minute later and add a late duplicate.
outbox.relay.poll-interval-msDefault none (application setting, 500 in this lesson)
What it does
Pause between relay polls. It is the typical extra latency between commit and publication when the system is idle.
When to change it
Lower it for faster events at the cost of more queries against the database. If you need tens of milliseconds, move to CDC instead of polling harder.
relay batch size (LIMIT)Default none (application setting, 100 in this lesson)
What it does
Maximum rows read and published per poll, and so the upper bound on how long the relay's transaction stays open.
When to change it
Raise it when a backlog drains too slowly. Keep batch size times per-send time well under any database idle-transaction timeout.
published-row retentionDefault none (application setting, 3 days in this lesson)
What it does
How long published rows are kept before the cleanup job deletes them.
When to change it
Long enough to answer 'was this event sent?' during an incident, short enough that the table and its indexes stay small.

Hands-on exercise

Break a dual write, then fix it with an outbox

See an order committed without its event, then see the outbox publish it after a crash, and count the duplicates the relay produces when it fails between sending and marking.

Steps · 0 of 5 done

Take it further in the lab: Transactional outbox

What can go wrong

Marking the row before the broker acknowledges
Symptom
Every outbox row has published_at set, yet a few orders have no record on the topic and stay PENDING.
Cause
The relay calls send and updates the row without waiting for the returned future. A send that later fails (leader election, timeout, record too large) is never retried because the row already looks published.
Fix
Wait for the acknowledgement (send(...).get(timeout) or a completion callback that does the marking) and mark only on success.
Consumers that are not idempotent
Symptom
A customer is charged twice for one order shortly after a deploy or restart of the Order Service.
Cause
The relay was stopped between the acknowledgement and the mark and republished the row. Producer idempotence does not cover this: it is a new send, not a retry of the old one.
Fix
Carry the outbox row's id as the event id and have each consumer record processed ids in the same transaction as its effect, or make the effect naturally idempotent (a unique key on order_id).
Events of one order out of order
Symptom
A consumer receives OrderCancelled before OrderCreated for the same order id, in the same partition.
Cause
Two relay instances published concurrently (for example with SKIP LOCKED), or the relay continued the batch after a failed send, or the key is not the aggregate id.
Fix
One active relay per outbox (advisory lock, leader election, or CDC), stop the batch on the first failure, and key by aggregate id.
The outbox table grows without bound
Symptom
Order placement gets slower over weeks. The outbox table and its indexes are larger than the orders table.
Cause
Published rows are never deleted, or the relay's query scans published rows because there is no index on the unpublished ones.
Fix
Add a partial index on unpublished rows and a scheduled, batched delete of old published rows. With CDC, delete the row in the inserting transaction.
A poison row stops the relay
Symptom
Unpublished rows pile up behind one row. The relay log repeats the same error on every poll.
Cause
One row can never be published (payload larger than max.request.size or the topic's max.message.bytes, or a topic that does not exist). Stopping at the first failure is right for order, but it blocks everything behind a permanent failure.
Fix
Alert on the age of the oldest unpublished row. Validate payload size when writing the row. For a permanent failure, move the row to a quarantine table by a deliberate, logged decision, because skipping it breaks that aggregate's order.
Writing the outbox row outside the transaction
Symptom
Occasional orders without events, even though an outbox is in place.
Cause
The outbox insert runs on a different connection or in a REQUIRES_NEW transaction, or in an @Async method or an after-commit hook, so it commits separately from the order row.
Fix
Insert both rows through the same transaction manager and connection, inside one @Transactional boundary, and test it by forcing a rollback after the outbox insert.

How to troubleshoot

Orders are stuck in PENDING and no downstream service reacts

Check

SELECT count(*) AS pending, min(created_at) AS oldest FROM outbox_event WHERE published_at IS NULL;
How to read it
A growing count with an old oldest means the relay is not publishing: it is down, cannot get the advisory lock, cannot reach Kafka, or is stuck on one row. A count near zero means the events were published and the problem is on the consuming side.
Events are published but the Payment Service is behind

Check

kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group payment-service
How to read it
Compare CURRENT-OFFSET with LOG-END-OFFSET per partition of orders.events. Lag that keeps growing on every partition means the consumer is slow or stopped. Lag on one partition suggests one record that keeps failing.
Suspected duplicate events on the topic

Check

kafka-console-consumer.sh --bootstrap-server broker:9092 --topic orders.events --from-beginning --property print.key=true --property print.headers=true
How to read it
Two records with the same id header are a relay republication, which is expected occasionally. Many of them point to the relay failing after every send, for example a failing UPDATE. Recent 4.x releases deprecate --property here in favour of --formatter-property.
The relay logs send timeouts

Check

The exception class in the relay log, for example `org.apache.kafka.common.errors.TimeoutException` or `RecordTooLargeException`
How to read it
A timeout on every row means the brokers are unreachable or the partition has too few in-sync replicas for acks=all: events will drain when that is fixed. RecordTooLargeException on one row is permanent and needs a decision about that row.

Key takeaways

  • A database write and a Kafka send cannot be made atomic. The outbox turns them into one local transaction plus a retried copy.
  • The event row is written in the same transaction as the business row. Anything else is still a dual write.
  • The relay sends first and marks second, so publication is at-least-once. Duplicates are expected and consumers must deduplicate by event id.
  • Producer idempotence handles the producer's retries of one send. It does not handle the relay sending a row again.
  • Order per aggregate needs the aggregate id as key, a single active relay, and stopping at the first failed send.
  • Monitor the age of the oldest unpublished row and clean up published rows.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

After a rolling deploy of the Order Service, support reports 14 customers charged twice. The topic orders.events contains two OrderCreated records for each of those orders, with identical id headers, a few seconds apart. The producer has enable.idempotence=true and acks=all.

What is the right fix?

Scenario 2 of 3

To drain a backlog faster, a team runs three relay instances that each select 100 unpublished rows with FOR UPDATE SKIP LOCKED and publish them. Inventory then reports that for some orders it received OrderCancelled before OrderCreated, both in the same partition.

What explains the reordering?

Scenario 3 of 3

A relay remembers the highest seq it has published and polls with WHERE seq > :last ORDER BY seq. Under load, about one order in 50,000 never gets its event. The missing rows exist in outbox_event with a seq lower than the relay's cursor.

Why were those rows skipped?

References

References: Apache Kafka 4.1: producer configuration reference (acks, enable.idempotence, delivery.timeout.ms) ↗ · Spring for Apache Kafka 3.3 reference: Transactions (synchronisation with other transaction managers) ↗ · Spring for Apache Kafka 3.3 reference: Sending messages (KafkaTemplate) ↗ · Debezium documentation: Outbox event router (outbox table columns) ↗ · PostgreSQL documentation: SELECT, the locking clause and SKIP LOCKED ↗ · PostgreSQL documentation: advisory lock functions ↗ · Chris Richardson, microservices.io: Transactional outbox pattern ↗ · Chris Richardson, microservices.io: Polling publisher pattern ↗ · Report a problem