Skip to content

BuildBreak lab 12 of 15

Transactional outbox

Save an order and publish its event as two separate writes, lose one of them in a crash, then fix it with an outbox table and a relay, and absorb the duplicate the relay can produce.

Advanced30 min

Read first: Kafka transactions and your database · Transactional outbox · The idempotent consumer pattern

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
orders: 3 partitions, RF 3
Producers
none
Consumer groups
payment-service (1)
Guided run
9 stages: build → break → observe → fix → retest

1

Business scenario

The order service saves each order in its PostgreSQL database and then publishes OrderCreated to the orders topic, where the payment service picks it up. Support has two kinds of ticket this month. Some customers have a confirmed order that was never charged and never shipped. Others were charged for an order that the order service has no record of. Both kinds started on days when the order service was restarted under load.

2

Starting architecture

Drawing diagram…

A customer places an order with the order service. The order service first inserts and commits a row in the orders table, and second sends an OrderCreated event to the orders topic. The payment service polls the topic and inserts a row into its payments table. Nothing ties the database commit and the Kafka send together.

flowchart LR
  C["Customer"] -->|"place order"| OS["Order service"]
  OS -->|"1 insert and commit"| DB[("orders table")]
  OS -->|"2 send OrderCreated"| T["Topic orders"]
  T -->|"poll"| PS["Payment service"]
  PS -->|"insert"| PDB[("payments table")]
The starting point is a dual write: the database commit and the Kafka send are two writes to two systems.

3

Expected behaviour

  • Every order row in the database has exactly one OrderCreated event on the topic, and every event belongs to an order that exists.
  • The order service and the relay can crash at any point without an order losing its event.
  • An event that is published twice has one effect in the payment service.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 9

Take orders with a dual write

Do this

  1. Turn on Customers placing orders (2 orders per second) and press play.
  2. Watch the orders table in the database panel and the orders topic fill at the same pace.

Look for

  • Every order row says "on the topic". "Orders with no event" and "Events with no order" are both 0.
  • The payment service inserts one payments row per event.

Not done yet. The simulator checks this for you:ordersInDb >= 8 && rows("payments") >= 6 && missingEvents == 0 && phantomEvents == 0

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

2 orders per second.

Order service database

simulated

Dual write: the order row is committed first, then the event is sent to Kafka. Two systems, two separate writes.

Write path
commit → send
Order rows
0
Outbox rows unpublished
0
Relay
not used

orders (newest first, problems pinned on top)

orderrowOrderCreated

Simulated metrics

computed by the simulation
Orders in the database
0
Events published
0
Orders with no event
0
Events with no order
0
Events the relay delivered after a crash
0
Duplicate events published by the relay
0
Duplicates applied by the payment service
0
Duplicates rejected by the payment service
0
Payment rows
0
Payment service lag
0records

Database

simulated · available
payments0 rowsno unique constraint on event_id
Rows of table payments: flagged rows first, then the newest
idevent_idvaluewrittennote
no rows yet

Partition logs

simulated records
  • orders-0leader b1ISR 1,2,3HW 0end 0

    empty

    payment-service: committed none, lag 0

  • orders-1leader b2ISR 2,3,1HW 0end 0

    empty

    payment-service: committed none, lag 0

  • orders-2leader b3ISR 3,1,2HW 0end 0

    empty

    payment-service: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders (3 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.