BuildBreak lab 12 of 15
Transactional outbox
Save an order and publish its event as two separate writes, lose one of them in a crash, then fix it with an outbox table and a relay, and absorb the duplicate the relay can produce.
Read first: Kafka transactions and your database · Transactional outbox · The idempotent consumer pattern
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- orders: 3 partitions, RF 3
- Producers
- none
- Consumer groups
- payment-service (1)
- Guided run
- 9 stages: build → break → observe → fix → retest
1
Business scenario
The order service saves each order in its PostgreSQL database and then publishes OrderCreated to the orders topic, where the payment service picks it up. Support has two kinds of ticket this month. Some customers have a confirmed order that was never charged and never shipped. Others were charged for an order that the order service has no record of. Both kinds started on days when the order service was restarted under load.
2
Starting architecture
Drawing diagram…
A customer places an order with the order service. The order service first inserts and commits a row in the orders table, and second sends an OrderCreated event to the orders topic. The payment service polls the topic and inserts a row into its payments table. Nothing ties the database commit and the Kafka send together.
flowchart LR
C["Customer"] -->|"place order"| OS["Order service"]
OS -->|"1 insert and commit"| DB[("orders table")]
OS -->|"2 send OrderCreated"| T["Topic orders"]
T -->|"poll"| PS["Payment service"]
PS -->|"insert"| PDB[("payments table")]3
Expected behaviour
- Every order row in the database has exactly one
OrderCreatedevent on the topic, and every event belongs to an order that exists. - The order service and the relay can crash at any point without an order losing its event.
- An event that is published twice has one effect in the payment service.
4 · Build → break → observe → fix → retest
Guided run
Take orders with a dual write
Do this
- Turn on Customers placing orders (2 orders per second) and press play.
- Watch the orders table in the database panel and the
orderstopic fill at the same pace.
Look for
- Every order row says "on the topic". "Orders with no event" and "Events with no order" are both 0.
- The payment service inserts one
paymentsrow per event.
Not done yet. The simulator checks this for you:ordersInDb >= 8 && rows("payments") >= 6 && missingEvents == 0 && phantomEvents == 0
Simulator
Simulation · not a real clusterControls for this stage
2 orders per second.
Order service database
simulatedDual write: the order row is committed first, then the event is sent to Kafka. Two systems, two separate writes.
- Write path
- commit → send
- Order rows
- 0
- Outbox rows unpublished
- 0
- Relay
- not used
orders (newest first, problems pinned on top)
| order | row | OrderCreated |
|---|
Simulated metrics
computed by the simulation- Orders in the database
- 0
- Events published
- 0
- Orders with no event
- 0
- Events with no order
- 0
- Events the relay delivered after a crash
- 0
- Duplicate events published by the relay
- 0
- Duplicates applied by the payment service
- 0
- Duplicates rejected by the payment service
- 0
- Payment rows
- 0
- Payment service lag
- 0records
Database
simulated · available| id | event_id | value | written | note |
|---|---|---|---|---|
| no rows yet | ||||
Partition logs
simulated records- orders-0leader b1ISR 1,2,3HW 0end 0
empty
payment-service: committed none, lag 0
- orders-1leader b2ISR 2,3,1HW 0end 0
empty
payment-service: committed none, lag 0
- orders-2leader b3ISR 3,1,2HW 0end 0
empty
payment-service: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders (3 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.