Path 7 · lesson 1 of 8
Transactional outbox
Write the event to a table in the same database transaction as the business row, then let a relay publish it to Kafka. No event is lost, and none is published for a rolled-back change.
What you will learn
- Explain why writing to a database and to Kafka from one method cannot be made atomic, and what each failure order leaves behind
- Design an outbox table and write the event row in the same local transaction as the business row
- Implement a polling relay in Spring Boot that publishes outbox rows in order and marks them only after the broker acknowledges
- State what the pattern guarantees (at-least-once publication) and build the consumer-side deduplication that it requires
- Keep events for one aggregate in order, and keep the outbox table small
Before you start
- What a Kafka transaction does not cover
- Producer idempotence vs consumer idempotency
- Keys, partitions and ordering
- Local transactions in a relational database (commit, rollback)
The concept
The Order Service does two things when a customer checks out: it stores the order in its own database, and it tells the rest of the system by publishing OrderCreated to Kafka. Payment, Inventory and Notification all start from that event. The two writes go to two different systems, and there is no transaction that covers both. This is the dual-write problem.
The transactional outbox removes the second write from the request path. Instead of calling Kafka, the service inserts a row describing the event into an outbox_event table in the same database transaction as the order row. Either both rows commit or neither does. A separate process, the message relay, later reads the outbox and publishes each row to Kafka.
- Outbox table: one row per event, in the service's own database: an event id, the aggregate it belongs to, the event type and the payload.
- Business transaction:
INSERT INTO ordersandINSERT INTO outbox_event, one commit. - Relay: publishes rows to Kafka after they commit. It is either a polling publisher (a loop that queries for unpublished rows, built in this lesson) or log-based change data capture (a tool that reads the database's transaction log, covered in the next lesson).
- Idempotent consumers: the relay can publish a row twice, so every consumer must recognise an event it has already handled.
Advantages
- The business change and its event are atomic, using only a local database transaction
- Placing an order does not depend on Kafka being reachable: events queue in the table during a broker outage
- No two-phase commit and no Kafka transaction spanning a database
- Events for one aggregate are published in the order they were committed
- The outbox table is an inspectable record of what was and was not sent
Disadvantages
- Publication is at-least-once, so every consumer needs deduplication
- Events arrive after a delay: the poll interval, or the CDC lag
- Every business transaction pays for one more insert
- A relay is one more thing to run, monitor and alert on
- The table grows without cleanup, and a stuck relay silently stops all downstream work
- Developers must remember to write the outbox row: nothing forces it
Outbox pattern
Simulation, not a real clusterThe animated lesson shows the same dual write failing and the outbox closing the gap, step by step.
Step 1 · The pattern
Save it, then tell everyone
An order comes in. The service saves it to its database, then publishes an event so shipping, email and analytics can react. Two writes, to two systems.
- order request
- done
Why it exists
Take the obvious implementation: orderRepository.save(order) followed by kafkaTemplate.send(...). Whichever order you choose, there is a moment between the two calls where a crash, a timeout or a rollback leaves the two systems disagreeing.
| Order of the two writes | What goes wrong | What the business sees |
|---|---|---|
| Commit to the database, then send to Kafka | The process dies, or the send fails, after the commit | An order that is never paid, never reserved and never confirmed. Nothing retries it, because nothing knows the event is missing. |
| Send to Kafka, then commit to the database | The commit fails (constraint violation, deadlock, lost connection) | A ghost event: Payment charges the customer for an order id that does not exist. |
| Send inside the database transaction, before the commit | The same as the row above: a Kafka send cannot be rolled back by the database | A ghost event again. send is also asynchronous, so the commit can succeed while the send later fails. |
| Kafka transaction around both | A Kafka transaction covers writes to Kafka partitions and consumer offset commits, and nothing else | Two separate commits that can still disagree. Spring can synchronise them, but that only shrinks the window. |
Spring for Apache Kafka can synchronise a Kafka transaction with a database transaction manager. The reference is explicit about what that means: the database transaction commits first and the Kafka transaction after it, and if the second commit fails the exception is thrown to the caller, who must take remedial action. It is a best-effort ordering of two commits. A crash between them still loses the event.
A distributed transaction (XA, two-phase commit) would be the textbook answer, but Kafka does not take part in XA transactions, and two-phase commit ties the availability of checkout to the availability of every participant. The outbox takes a different route: use the one system that already offers atomicity, the service's own database, for both facts. Publishing then becomes a separate problem: copying committed rows to Kafka with retries. That problem is easy to make reliable.
How it works inside
The write side
The outbox row is ordinary data. It is inserted with the same connection and in the same transaction as the business row, so the database's commit makes both durable together, and a rollback removes both. The row carries everything the relay needs to build a Kafka record: a unique event id (so consumers can deduplicate), the aggregate type (which decides the topic), the aggregate id (which becomes the record key) and the serialised payload.
The relay: a polling publisher
- Select a batch of rows where
published_at IS NULL, ordered by an insertion sequence. - For each row, send a record to Kafka with
key = aggregateidand wait for the broker's acknowledgement. - Only after the acknowledgement, set
published_aton that row. - Commit, sleep for the poll interval, repeat.
Steps 2 and 3 are themselves a dual write: Kafka first, then the database. The relay cannot make them atomic either. What it can do is choose the safe order. Marking a row before sending could lose the event. Sending before marking can only repeat it. That choice is where at-least-once comes from.
| Crash point | Resulting state | Recovery |
|---|---|---|
| Order Service, before its commit | No order row, no outbox row | Nothing to recover. The client got an error and can retry the request. |
| Order Service, after its commit | Order row and outbox row both exist, unpublished | The relay finds the row on its next poll and publishes it. No event is lost. |
| Relay, after reading the batch, before sending | Rows still unpublished | The next poll reads the same rows. No duplicate. |
| Relay, after Kafka acknowledged, before marking the row | Record is in Kafka, row still says unpublished | The next poll sends it again: a duplicate with the same event id, which consumers skip. |
| Kafka unavailable | Sends fail or time out, rows stay unpublished and accumulate | Orders are still accepted. When the brokers return, the relay drains the backlog in order. |
| Database unavailable | No orders can be placed and the relay cannot read | Nothing is half-done. Both resume when the database returns. |
| Consumer, after its own commit, before the offset commit | Effect applied, offset not advanced | The record is redelivered, and the consumer's processed-event check turns it into a no-op. |
Ordering per aggregate
Kafka orders records only within a partition, so the relay uses the aggregate id (orderId) as the key: all events of one order land in one partition. That is necessary but not sufficient. The relay must also hand them to the producer in commit order, and three things break that:
- Several relay instances reading at once. With
FOR UPDATE SKIP LOCKED, instance B skips the rows instance A has locked and may publish a later event of the same order first. The code below allows only one active relay at a time, using a PostgreSQL advisory lock. - Continuing past a failed send. If row 3 fails and the relay goes on to row 4 of the same order, the retry of row 3 arrives after it. The relay stops the batch at the first failure.
- A cursor on the sequence (
WHERE seq > :lastSeen). Sequence values are assigned at insert time, and transactions commit in a different order, so a row with a lowerseqcan become visible after the cursor has passed it. That row is then never published. Selecting onpublished_at IS NULLcannot skip a row.
Two transactions that change the same order are serialised by the row lock on that order, so their outbox rows get increasing seq values in commit order. Events of different orders may be published in a slightly different order than they committed, which no consumer should depend on anyway.
Cleanup
Published rows have no further use to the relay. Delete them in small batches on a schedule, keeping a few days for debugging. A partial index on unpublished rows keeps the relay's query fast regardless of how many published rows remain. With a CDC relay the row can be deleted in the same transaction that inserts it, because the relay reads the transaction log and not the table.
Architecture
Drawing diagram…
Flowchart: the Order Service inserts into the orders table and the outbox table in one database transaction. An outbox relay polls unpublished rows, sends them to the orders.events topic keyed by order id, then marks them published. Payment, Inventory and Notification consume the topic, and Payment records processed event ids.
flowchart LR
Client["Checkout request"] --> OS["Order Service"]
subgraph DB["Order database, one local transaction"]
OT["orders table"]
OB["outbox_event table"]
end
OS -->|"INSERT order"| OT
OS -->|"INSERT event row"| OB
OB -->|"poll unpublished rows"| RL["Outbox relay"]
RL -->|"send key orderId, wait for ack"| T["Topic orders.events"]
RL -->|"mark published_at"| OB
T --> PS["Payment Service"]
T --> IS["Inventory Service"]
T --> NS["Notification Service"]
PS --> PE["processed_event table"]- The Order Service has no Kafka producer on the request path. Its only dependency for accepting an order is its own database.
- The relay is the single writer of
orders.events. One writer per aggregate type, sending in sequence order, is what keeps events of one order in order. - The arrow from the relay back to the outbox (
mark published_at) happens after the broker's acknowledgement. A crash between the two is the source of duplicates. - Each consumer keeps its own
processed_eventtable in its own database, written in the same transaction as its business effect.
Message flow
Drawing diagram…
Sequence diagram: the Order Service inserts the order and the outbox row and commits. The relay reads the row, sends it to Kafka and is acknowledged, then crashes before marking it. After restart it reads and sends the same row again and marks it. The Payment Service receives two copies with the same event id, processes the first and skips the second.
sequenceDiagram participant C as Client participant O as Order Service participant D as Order DB participant R as Outbox relay participant K as Kafka orders.events participant P as Payment Service C->>O: POST /orders O->>D: BEGIN O->>D: INSERT orders row O->>D: INSERT outbox_event row O->>D: COMMIT O-->>C: 201 Created, order PENDING R->>D: SELECT unpublished rows ORDER BY seq D-->>R: OrderCreated, event id e1 R->>K: send key o-1001, header id e1 K-->>R: ack Note over R: crash here, before marking the row R->>D: SELECT unpublished rows after restart D-->>R: OrderCreated e1 again R->>K: send key o-1001, header id e1 K-->>R: ack R->>D: UPDATE published_at for e1 K->>P: OrderCreated e1, first copy P->>P: insert e1 into processed_event, create payment K->>P: OrderCreated e1, second copy P->>P: e1 already processed, skip
- The client gets its response as soon as the local transaction commits. Publication happens afterwards, so the API is asynchronous from this point: the order is
PENDINGuntil later events arrive. - The duplicate is not a producer retry. It is a second, independent
sendof the same business event, which is why producer idempotence does not remove it. - Both copies carry the same event id in a header because the id is stored in the outbox row, not generated at send time.
In a real system
A typical order system uses one outbox table per service database. The Order Service writes OrderCreated, OrderConfirmed and OrderCancelled rows with aggregatetype = 'orders'. Its relay publishes them to orders.events, a topic with 12 partitions and replication factor 3, with the order id as key. The Payment Service has its own outbox for payments.events, and Inventory has one for inventory.events. Consumer groups are payment-service, inventory-service and notification-service, each reading orders.events independently.
| Decision | Choice in this system | Reason |
|---|---|---|
| Record key | aggregateid (order id) | All events of an order share a partition, so consumers see them in order |
| Event id | UUID stored in the outbox row, sent as header id | Stable across republication, so consumers can deduplicate |
| Event type | Column type, sent as header eventType | Consumers can route without parsing the payload |
| Payload | A complete, self-describing JSON event | Consumers must not need to call back to the Order Service to understand it |
| Relay | Polling publisher first, CDC when latency or load requires it | Polling needs no extra infrastructure and works on any database |
| Retention of published rows | Deleted after 3 days | Enough to investigate an incident without letting the table grow |
When to use it
- A service changes its own database and must publish an event about that change, and a missing or phantom event has a business cost (orders, payments, stock, account changes).
- The service must keep accepting writes while Kafka is unavailable.
- Consumers can be made idempotent, which they need to be for at-least-once consumption in any case.
- A delay of a few hundred milliseconds between commit and publication is acceptable.
When not to use it
- There is no database write to stay consistent with: a service that only transforms records from one topic to another should use Kafka transactions for read-process-write.
- The data is high-volume telemetry such as clickstream, where an occasional lost record is acceptable and an extra insert per event is not. Produce directly.
- The consumer needs the row state, not a business event (cache or search index updates). Plain change data capture on the table is simpler than writing outbox rows.
- The database cannot offer a multi-row transaction across the business table and the outbox table. The pattern depends entirely on that.
Java and Spring Boot
The Order Service on Spring Boot 3 with JdbcTemplate and PostgreSQL: the schema, the transactional write, a polling relay, and a consumer that deduplicates. Serializers are StringSerializer and StringDeserializer: the payload is stored as JSON in the outbox and sent unchanged.
-- Order Service databaseCREATE TABLE orders ( id varchar(36) PRIMARY KEY, customer_id varchar(36) NOT NULL, sku varchar(64) NOT NULL, quantity integer NOT NULL, total_cents bigint NOT NULL, status varchar(16) NOT NULL, created_at timestamptz NOT NULL DEFAULT now());CREATE TABLE outbox_event ( seq bigserial PRIMARY KEY, -- publication order id uuid NOT NULL UNIQUE, -- event id, used by consumers to deduplicate aggregatetype varchar(255) NOT NULL, -- decides the topic: 'orders' -> orders.events aggregateid varchar(255) NOT NULL, -- becomes the Kafka record key type varchar(255) NOT NULL, -- 'OrderCreated', 'OrderCancelled', ... payload jsonb NOT NULL, created_at timestamptz NOT NULL DEFAULT now(), published_at timestamptz -- NULL until the broker has acknowledged);-- The relay only ever looks at unpublished rows: keep that lookup small.CREATE INDEX outbox_unpublished ON outbox_event (seq) WHERE published_at IS NULL;-- Transaction 1 (request thread): the business row and the event row, one commit.-- BEGIN;-- INSERT INTO orders (...) VALUES (...);-- INSERT INTO outbox_event (id, aggregatetype, aggregateid, type, payload) VALUES (...);-- COMMIT;-- Transaction 2 (relay): read in order, send to Kafka, mark what was acknowledged.-- BEGIN;-- SELECT pg_try_advisory_xact_lock(7301);-- SELECT ... FROM outbox_event WHERE published_at IS NULL ORDER BY seq LIMIT 100;-- UPDATE outbox_event SET published_at = now() WHERE seq = ?; -- once per acknowledged row-- COMMIT;-- Cleanup job (scheduled, small batches):-- DELETE FROM outbox_event-- WHERE seq IN (SELECT seq FROM outbox_event-- WHERE published_at < now() - interval '3 days' LIMIT 5000);-- Payment Service database (a different database): the consumer's dedupe table.CREATE TABLE processed_event ( consumer varchar(64) NOT NULL, event_id uuid NOT NULL, processed_at timestamptz NOT NULL DEFAULT now(), PRIMARY KEY (consumer, event_id));CREATE TABLE payment ( order_id varchar(36) PRIMARY KEY, -- one payment per order amount_cents bigint NOT NULL, status varchar(16) NOT NULL);Why it is written this way
- The column names
id,aggregatetype,aggregateid,typeandpayloadmatch the defaults of Debezium's outbox event router, so the same table works when the polling relay is later replaced by CDC. seqgives the relay a publication order.idis a separate UUID because consumers need an identifier that means the same thing in every service.- The partial index covers only unpublished rows, so the relay's query cost does not depend on how many published rows are waiting for cleanup.
processed_eventlives in the consumer's database and is written in the same transaction as the consumer's effect. A dedupe store outside that transaction reintroduces a dual write.
package com.example.orders;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.ObjectMapper;import java.util.UUID;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.stereotype.Service;import org.springframework.transaction.annotation.Transactional;@Servicepublic class OrderService { public record PlaceOrder(String customerId, String sku, int quantity, long totalCents) {} public record OrderCreated(String orderId, String customerId, String sku, int quantity, long totalCents) {} private final JdbcTemplate jdbc; private final ObjectMapper json; public OrderService(JdbcTemplate jdbc, ObjectMapper json) { this.jdbc = jdbc; this.json = json; } @Transactional public String placeOrder(PlaceOrder cmd) { String orderId = UUID.randomUUID().toString(); jdbc.update(""" INSERT INTO orders (id, customer_id, sku, quantity, total_cents, status) VALUES (?, ?, ?, ?, ?, 'PENDING')""", orderId, cmd.customerId(), cmd.sku(), cmd.quantity(), cmd.totalCents()); var event = new OrderCreated(orderId, cmd.customerId(), cmd.sku(), cmd.quantity(), cmd.totalCents()); jdbc.update(""" INSERT INTO outbox_event (id, aggregatetype, aggregateid, type, payload) VALUES (?, 'orders', ?, 'OrderCreated', ?::jsonb)""", UUID.randomUUID(), orderId, toJson(event)); return orderId; // No KafkaTemplate here. If this method throws, both inserts roll back and no event exists. } private String toJson(Object event) { try { return json.writeValueAsString(event); } catch (JsonProcessingException e) { throw new IllegalStateException("Event cannot be serialised", e); } }}Why it is written this way
- Both inserts run on the connection bound to the
@Transactionalmethod, so one commit covers them. That single commit is the whole pattern. - The payload is serialised here, inside the transaction. A serialisation failure rolls the order back, which is better than discovering an unpublishable row in the relay.
- The event id is generated once, when the row is written. The relay never generates ids, so a republished row keeps the same id.
- The method returns while the order is still
PENDING. Callers learn the outcome later, by polling the order or from a notification.
package com.example.orders;import java.nio.charset.StandardCharsets;import java.util.List;import java.util.UUID;import java.util.concurrent.TimeUnit;import org.apache.kafka.clients.producer.ProducerRecord;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.scheduling.annotation.Scheduled;import org.springframework.stereotype.Component;import org.springframework.transaction.support.TransactionTemplate;@Componentpublic class OutboxRelay { private static final Logger log = LoggerFactory.getLogger(OutboxRelay.class); private static final long RELAY_LOCK = 7301L; private static final int BATCH_SIZE = 100; private record OutboxRow(long seq, UUID id, String aggregateType, String aggregateId, String type, String payload) {} private final JdbcTemplate jdbc; private final TransactionTemplate tx; private final KafkaTemplate<String, String> kafka; public OutboxRelay(JdbcTemplate jdbc, TransactionTemplate tx, KafkaTemplate<String, String> kafka) { this.jdbc = jdbc; this.tx = tx; this.kafka = kafka; } @Scheduled(fixedDelayString = "${outbox.relay.poll-interval-ms:500}") public void publishPending() { tx.executeWithoutResult(status -> { // Only one relay instance publishes at a time. The lock is released at commit or rollback. Boolean active = jdbc.queryForObject("SELECT pg_try_advisory_xact_lock(?)", Boolean.class, RELAY_LOCK); if (!Boolean.TRUE.equals(active)) { return; } List<OutboxRow> batch = jdbc.query(""" SELECT seq, id, aggregatetype, aggregateid, type, payload::text AS payload FROM outbox_event WHERE published_at IS NULL ORDER BY seq LIMIT ?""", (rs, n) -> new OutboxRow( rs.getLong("seq"), rs.getObject("id", UUID.class), rs.getString("aggregatetype"), rs.getString("aggregateid"), rs.getString("type"), rs.getString("payload")), BATCH_SIZE); for (OutboxRow row : batch) { var record = new ProducerRecord<String, String>( row.aggregateType() + ".events", row.aggregateId(), row.payload()); record.headers() .add("id", row.id().toString().getBytes(StandardCharsets.UTF_8)) .add("eventType", row.type().getBytes(StandardCharsets.UTF_8)); try { // Wait for the broker's acknowledgement before marking the row. kafka.send(record).get(10, TimeUnit.SECONDS); } catch (InterruptedException e) { Thread.currentThread().interrupt(); return; } catch (Exception e) { log.warn("Outbox row {} not published, stopping this batch: {}", row.seq(), e.toString()); return; // rows already marked are committed, the rest are retried on the next poll } jdbc.update("UPDATE outbox_event SET published_at = now() WHERE seq = ?", row.seq()); } }); } @Scheduled(cron = "0 15 * * * *") public void deletePublished() { int deleted = jdbc.update(""" DELETE FROM outbox_event WHERE seq IN (SELECT seq FROM outbox_event WHERE published_at < now() - interval '3 days' LIMIT 5000)"""); log.debug("Outbox cleanup removed {} rows", deleted); }}Why it is written this way
pg_try_advisory_xact_lockmakes the relay single-active across all instances of the service without a leader-election library. A second instance getsfalseand skips this round. On another database, use that database's equivalent or a lock row.- The relay waits for each acknowledgement and stops at the first failure. Sending the whole batch and waiting on all futures is faster, but a failure in the middle can then leave a later event of the same order published before an earlier one.
KafkaTemplate.sendreturns aCompletableFuture. Withoutget, the method would mark rows whose sends later fail. That is the original bug moved into the relay.- The row is marked only after the acknowledgement. A crash between the two repeats the send on the next poll, which is the at-least-once behaviour consumers are built for.
- The database transaction stays open while sending. The batch size and the 10 second wait bound how long. Keep the producer's
delivery.timeout.msin mind: a send whosegettimed out can still be delivered later, which is one more source of duplicates. @Scheduledneeds@EnableSchedulingon a configuration class. The cleanup deletes in bounded batches so it never holds a long lock.
package com.example.payments;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.ObjectMapper;import java.nio.charset.StandardCharsets;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.header.Header;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;@Componentpublic class OrderEventsListener { public record OrderCreated(String orderId, String customerId, String sku, int quantity, long totalCents) {} private final JdbcTemplate jdbc; private final ObjectMapper json; public OrderEventsListener(JdbcTemplate jdbc, ObjectMapper json) { this.jdbc = jdbc; this.json = json; } @KafkaListener(id = "payment-orders", topics = "orders.events", groupId = "payment-service") @Transactional public void onOrderEvent(ConsumerRecord<String, String> record) throws JsonProcessingException { if (!"OrderCreated".equals(header(record, "eventType"))) { return; } String eventId = header(record, "id"); // First writer wins. 0 rows means this event id was already processed by this service. int firstTime = jdbc.update(""" INSERT INTO processed_event (consumer, event_id) VALUES ('payment-service', ?::uuid) ON CONFLICT DO NOTHING""", eventId); if (firstTime == 0) { return; } OrderCreated event = json.readValue(record.value(), OrderCreated.class); jdbc.update(""" INSERT INTO payment (order_id, amount_cents, status) VALUES (?, ?, 'PENDING')""", event.orderId(), event.totalCents()); // The dedupe row and the payment row commit together when this method returns. // The listener container commits the offset afterwards. } private static String header(ConsumerRecord<String, String> record, String name) { Header h = record.headers().lastHeader(name); if (h == null) { throw new IllegalArgumentException("Missing header " + name); } return new String(h.value(), StandardCharsets.UTF_8); }}Why it is written this way
- The dedupe insert and the business insert share one database transaction. If the service crashes before the commit, neither exists and the redelivered record is processed normally. If it crashes after, the redelivery is skipped.
ON CONFLICT DO NOTHINGmakes the check atomic. A separateSELECTfollowed by anINSERTwould let two concurrent deliveries both pass the check.- The offset is committed by the container after the method returns, so this is at-least-once consumption made safe by the dedupe row.
- A missing header throws. Configure the container's
DefaultErrorHandlerwith a dead letter recoverer so a malformed record does not block the partition (see the error handling lessons).
Configuration
The relay's producer should favour durability and order over latency. Kafka defaults below are from the Kafka 4.1 producer configuration reference. The last three are application settings of the relay shown above, not Kafka settings.
acksDefault all
- What it does
- The leader waits for all in-sync replicas before acknowledging. With
min.insync.replicas=2and replication factor 3 on the topic, an acknowledged event survives one broker failure. - When to change it
- Leave at
all. The relay marks a row as published on the strength of this acknowledgement, so a weaker setting turns the outbox back into a way to lose events.
enable.idempotenceDefault true
- What it does
- The broker discards duplicates caused by the producer retrying a batch within its session, and keeps per-partition order across those retries.
- When to change it
- Leave on. It does not deduplicate a row the relay sends twice after a crash. That is the consumer's job.
max.in.flight.requests.per.connectionDefault 5
- What it does
- Unacknowledged requests per connection. With idempotence enabled, order is preserved for any value up to 5.
- When to change it
- No need to lower it while idempotence is on. If idempotence is ever disabled, values above 1 allow reordering on retry.
delivery.timeout.msDefault 120000 (2 minutes)
- What it does
- Upper bound on the time from
send()returning to the producer reporting success or failure, including retries. - When to change it
- Consider lowering it towards the relay's own wait so that a send the relay has given up on does not still succeed a minute later and add a late duplicate.
outbox.relay.poll-interval-msDefault none (application setting, 500 in this lesson)
- What it does
- Pause between relay polls. It is the typical extra latency between commit and publication when the system is idle.
- When to change it
- Lower it for faster events at the cost of more queries against the database. If you need tens of milliseconds, move to CDC instead of polling harder.
relay batch size (LIMIT)Default none (application setting, 100 in this lesson)
- What it does
- Maximum rows read and published per poll, and so the upper bound on how long the relay's transaction stays open.
- When to change it
- Raise it when a backlog drains too slowly. Keep batch size times per-send time well under any database idle-transaction timeout.
published-row retentionDefault none (application setting, 3 days in this lesson)
- What it does
- How long published rows are kept before the cleanup job deletes them.
- When to change it
- Long enough to answer 'was this event sent?' during an incident, short enough that the table and its indexes stay small.
Hands-on exercise
Break a dual write, then fix it with an outbox
See an order committed without its event, then see the outbox publish it after a crash, and count the duplicates the relay produces when it fails between sending and marking.
Steps · 0 of 5 done
What you should see
With the direct send, one order exists with no event and stays PENDING forever. With the outbox, the same crash loses nothing: the event appears after the relay's next poll. The relay crash puts two records with the same event id on the topic. Without deduplication that creates two payments. With it, one payment and one skipped record.
What can go wrong
Marking the row before the broker acknowledges
- Symptom
- Every outbox row has
published_atset, yet a few orders have no record on the topic and stay PENDING. - Cause
- The relay calls
sendand updates the row without waiting for the returned future. A send that later fails (leader election, timeout, record too large) is never retried because the row already looks published. - Fix
- Wait for the acknowledgement (
send(...).get(timeout)or a completion callback that does the marking) and mark only on success.
Consumers that are not idempotent
- Symptom
- A customer is charged twice for one order shortly after a deploy or restart of the Order Service.
- Cause
- The relay was stopped between the acknowledgement and the mark and republished the row. Producer idempotence does not cover this: it is a new send, not a retry of the old one.
- Fix
- Carry the outbox row's id as the event id and have each consumer record processed ids in the same transaction as its effect, or make the effect naturally idempotent (a unique key on
order_id).
Events of one order out of order
- Symptom
- A consumer receives
OrderCancelledbeforeOrderCreatedfor the same order id, in the same partition. - Cause
- Two relay instances published concurrently (for example with
SKIP LOCKED), or the relay continued the batch after a failed send, or the key is not the aggregate id. - Fix
- One active relay per outbox (advisory lock, leader election, or CDC), stop the batch on the first failure, and key by aggregate id.
The outbox table grows without bound
- Symptom
- Order placement gets slower over weeks. The outbox table and its indexes are larger than the orders table.
- Cause
- Published rows are never deleted, or the relay's query scans published rows because there is no index on the unpublished ones.
- Fix
- Add a partial index on unpublished rows and a scheduled, batched delete of old published rows. With CDC, delete the row in the inserting transaction.
A poison row stops the relay
- Symptom
- Unpublished rows pile up behind one row. The relay log repeats the same error on every poll.
- Cause
- One row can never be published (payload larger than
max.request.sizeor the topic'smax.message.bytes, or a topic that does not exist). Stopping at the first failure is right for order, but it blocks everything behind a permanent failure. - Fix
- Alert on the age of the oldest unpublished row. Validate payload size when writing the row. For a permanent failure, move the row to a quarantine table by a deliberate, logged decision, because skipping it breaks that aggregate's order.
Writing the outbox row outside the transaction
- Symptom
- Occasional orders without events, even though an outbox is in place.
- Cause
- The outbox insert runs on a different connection or in a
REQUIRES_NEWtransaction, or in an@Asyncmethod or an after-commit hook, so it commits separately from the order row. - Fix
- Insert both rows through the same transaction manager and connection, inside one
@Transactionalboundary, and test it by forcing a rollback after the outbox insert.
How to troubleshoot
Orders are stuck in PENDING and no downstream service reacts
Check
SELECT count(*) AS pending, min(created_at) AS oldest FROM outbox_event WHERE published_at IS NULL;
- How to read it
- A growing count with an old
oldestmeans the relay is not publishing: it is down, cannot get the advisory lock, cannot reach Kafka, or is stuck on one row. A count near zero means the events were published and the problem is on the consuming side.
Events are published but the Payment Service is behind
Check
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group payment-service
- How to read it
- Compare CURRENT-OFFSET with LOG-END-OFFSET per partition of
orders.events. Lag that keeps growing on every partition means the consumer is slow or stopped. Lag on one partition suggests one record that keeps failing.
Suspected duplicate events on the topic
Check
kafka-console-consumer.sh --bootstrap-server broker:9092 --topic orders.events --from-beginning --property print.key=true --property print.headers=true
- How to read it
- Two records with the same
idheader are a relay republication, which is expected occasionally. Many of them point to the relay failing after every send, for example a failingUPDATE. Recent 4.x releases deprecate--propertyhere in favour of--formatter-property.
The relay logs send timeouts
Check
The exception class in the relay log, for example `org.apache.kafka.common.errors.TimeoutException` or `RecordTooLargeException`
- How to read it
- A timeout on every row means the brokers are unreachable or the partition has too few in-sync replicas for
acks=all: events will drain when that is fixed.RecordTooLargeExceptionon one row is permanent and needs a decision about that row.
Key takeaways
- A database write and a Kafka send cannot be made atomic. The outbox turns them into one local transaction plus a retried copy.
- The event row is written in the same transaction as the business row. Anything else is still a dual write.
- The relay sends first and marks second, so publication is at-least-once. Duplicates are expected and consumers must deduplicate by event id.
- Producer idempotence handles the producer's retries of one send. It does not handle the relay sending a row again.
- Order per aggregate needs the aggregate id as key, a single active relay, and stopping at the first failed send.
- Monitor the age of the oldest unpublished row and clean up published rows.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
After a rolling deploy of the Order Service, support reports 14 customers charged twice. The topic orders.events contains two OrderCreated records for each of those orders, with identical id headers, a few seconds apart. The producer has enable.idempotence=true and acks=all.
What is the right fix?
Scenario 2 of 3
To drain a backlog faster, a team runs three relay instances that each select 100 unpublished rows with FOR UPDATE SKIP LOCKED and publish them. Inventory then reports that for some orders it received OrderCancelled before OrderCreated, both in the same partition.
What explains the reordering?
Scenario 3 of 3
A relay remembers the highest seq it has published and polls with WHERE seq > :last ORDER BY seq. Under load, about one order in 50,000 never gets its event. The missing rows exist in outbox_event with a seq lower than the relay's cursor.
Why were those rows skipped?
References
References: Apache Kafka 4.1: producer configuration reference (acks, enable.idempotence, delivery.timeout.ms) ↗ · Spring for Apache Kafka 3.3 reference: Transactions (synchronisation with other transaction managers) ↗ · Spring for Apache Kafka 3.3 reference: Sending messages (KafkaTemplate) ↗ · Debezium documentation: Outbox event router (outbox table columns) ↗ · PostgreSQL documentation: SELECT, the locking clause and SKIP LOCKED ↗ · PostgreSQL documentation: advisory lock functions ↗ · Chris Richardson, microservices.io: Transactional outbox pattern ↗ · Chris Richardson, microservices.io: Polling publisher pattern ↗ · Report a problem