BuildBreak lab 14 of 15
Kafka transaction
A service reads orders and writes to two topics. Crash it between the two sends, make the sends and the offset one Kafka transaction, fence a zombie, then find where the transaction stops.
Read first: Kafka transactions and the read-process-write loop · Kafka transactions in Spring Boot · Kafka transactions and your database · Effectively-once processing for payments and orders
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- orders: 1 partition, RF 3 · payments: 1 partition, RF 3 · audit: 1 partition, RF 3
- Producers
- checkout
- Consumer groups
- order-processor (1), payment-gateway (1), audit-store (1)
- Guided run
- 11 stages: build → break → observe → fix → retest
1
Business scenario
The order processor reads OrderPlaced from orders. For each order it sends a PaymentRequested record to payments, which the payment gateway turns into a card charge, and an AuditEntry record to audit, which the audit store keeps for the finance team. Finance reports two problems after a week with several restarts: charges that have no audit entry, and customers charged twice for one order. The team wants every order to produce both records exactly once.
2
Starting architecture
Drawing diagram…
The checkout producer writes OrderPlaced events to the orders topic. The order processor polls it and, for each order, first sends a PaymentRequested record to the payments topic, second sends an AuditEntry record to the audit topic, and third commits its offset. The payment gateway reads the payments topic and writes to a charges table. The audit store reads the audit topic.
flowchart LR
CO["Checkout producer"] -->|"OrderPlaced"| O["Topic orders"]
O -->|"poll"| OP["Order processor"]
OP -->|"1 send PaymentRequested"| P["Topic payments"]
OP -->|"2 send AuditEntry"| A["Topic audit"]
OP -->|"3 commit offset"| OFF["Group offsets"]
P --> PG["Payment gateway"]
A --> AS["Audit store"]
PG --> CH[("charges table")]3
Expected behaviour
- Every order leads to exactly one visible record on
paymentsand one onaudit, or to neither while it is being retried. - The processor can crash or freeze at any point without the payment gateway charging twice.
- An instance that the group has replaced cannot write output any more.
- Anything the handler does outside Kafka, such as a database insert, stays correct when the handler runs twice.
4 · Build → break → observe → fix → retest
Guided run
Process orders without transactions
Do this
- Turn on Checkout publishing (2 orders per second) and press play.
- Watch the output table: every order gets a record on
paymentsand one onaudit.
Look for
- "Charges made" and "Audit entries" grow together. "Partial writes" is 0.
- The transaction column says "none": each send is an ordinary write that consumers can read at once.
Not done yet. The simulator checks this for you:inserted("charges") >= 6 && inserted("audit_entries") >= 6 && partialWrites == 0
Simulator
Simulation · not a real clusterControls for this stage
2 orders per second.
Transactions on the output topics
simulatedA Kafka transaction covers the records this service writes to Kafka and its input offset. It never covers a database or any other external system.
- Transactional producer
- off
- Downstream reads (isolation.level)
- uncommitted
- Open transactions
- 0
- Aborted records in the log
- 0
Newest records on payments and audit
| topic | offset | order | transaction | visible downstream |
|---|
Simulated metrics
computed by the simulation- Charges made
- 0
- Audit entries
- 0
- Double charges
- 0
- Partial writes
- 0
- Transactions committed
- 0
- Transactions aborted
- 0
- Aborted records read downstream
- 0
- Aborted records skipped downstream
- 0
- Zombie producers fenced
- 0
- Ledger rows left by aborted transactions
- 0
- Duplicate ledger rows
- 0
- Duplicate ledger inserts rejected
- 0
Database
simulated · available| id | order_id | value | written | note |
|---|---|---|---|---|
| no rows yet | ||||
| id | order_id | value | written | note |
|---|---|---|---|---|
| no rows yet | ||||
Consumer group order-processor
stablegeneration 1lag 0RangeAssignor (eager)commit after each recordmax.poll.records 5
- processor-1runningprocessed 0orders-0
| Partition | Owner | Committed | Lag |
|---|---|---|---|
| orders-0 | processor-1 | none | 0 |
Partition logs
simulated records- orders-0leader b1ISR 1,2,3HW 0end 0
empty
order-processor: committed none, lag 0
- payments-0leader b1ISR 1,2,3HW 0end 0
empty
payment-gateway: committed none, lag 0
- audit-0leader b1ISR 1,2,3HW 0end 0
empty
audit-store: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders (1 partitions, RF 3), payments (1 partitions, RF 3), audit (1 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.