Skip to content

BuildBreak lab 14 of 15

Kafka transaction

A service reads orders and writes to two topics. Crash it between the two sends, make the sends and the offset one Kafka transaction, fence a zombie, then find where the transaction stops.

Advanced35 min

Read first: Kafka transactions and the read-process-write loop · Kafka transactions in Spring Boot · Kafka transactions and your database · Effectively-once processing for payments and orders

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
orders: 1 partition, RF 3 · payments: 1 partition, RF 3 · audit: 1 partition, RF 3
Producers
checkout
Consumer groups
order-processor (1), payment-gateway (1), audit-store (1)
Guided run
11 stages: build → break → observe → fix → retest

1

Business scenario

The order processor reads OrderPlaced from orders. For each order it sends a PaymentRequested record to payments, which the payment gateway turns into a card charge, and an AuditEntry record to audit, which the audit store keeps for the finance team. Finance reports two problems after a week with several restarts: charges that have no audit entry, and customers charged twice for one order. The team wants every order to produce both records exactly once.

2

Starting architecture

Drawing diagram…

The checkout producer writes OrderPlaced events to the orders topic. The order processor polls it and, for each order, first sends a PaymentRequested record to the payments topic, second sends an AuditEntry record to the audit topic, and third commits its offset. The payment gateway reads the payments topic and writes to a charges table. The audit store reads the audit topic.

flowchart LR
  CO["Checkout producer"] -->|"OrderPlaced"| O["Topic orders"]
  O -->|"poll"| OP["Order processor"]
  OP -->|"1 send PaymentRequested"| P["Topic payments"]
  OP -->|"2 send AuditEntry"| A["Topic audit"]
  OP -->|"3 commit offset"| OFF["Group offsets"]
  P --> PG["Payment gateway"]
  A --> AS["Audit store"]
  PG --> CH[("charges table")]
Consume, process, produce: two sends and an offset commit, which start out as three separate writes.

3

Expected behaviour

  • Every order leads to exactly one visible record on payments and one on audit, or to neither while it is being retried.
  • The processor can crash or freeze at any point without the payment gateway charging twice.
  • An instance that the group has replaced cannot write output any more.
  • Anything the handler does outside Kafka, such as a database insert, stays correct when the handler runs twice.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 11

Process orders without transactions

Do this

  1. Turn on Checkout publishing (2 orders per second) and press play.
  2. Watch the output table: every order gets a record on payments and one on audit.

Look for

  • "Charges made" and "Audit entries" grow together. "Partial writes" is 0.
  • The transaction column says "none": each send is an ordinary write that consumers can read at once.

Not done yet. The simulator checks this for you:inserted("charges") >= 6 && inserted("audit_entries") >= 6 && partialWrites == 0

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

2 orders per second.

Transactions on the output topics

simulated

A Kafka transaction covers the records this service writes to Kafka and its input offset. It never covers a database or any other external system.

Transactional producer
off
Downstream reads (isolation.level)
uncommitted
Open transactions
0
Aborted records in the log
0

Newest records on payments and audit

topicoffsetordertransactionvisible downstream

Simulated metrics

computed by the simulation
Charges made
0
Audit entries
0
Double charges
0
Partial writes
0
Transactions committed
0
Transactions aborted
0
Aborted records read downstream
0
Aborted records skipped downstream
0
Zombie producers fenced
0
Ledger rows left by aborted transactions
0
Duplicate ledger rows
0
Duplicate ledger inserts rejected
0

Database

simulated · available
charges0 rowsno unique constraint on order_id
Rows of table charges: flagged rows first, then the newest
idorder_idvaluewrittennote
no rows yet
ledger0 rowsno unique constraint on order_id
Rows of table ledger: flagged rows first, then the newest
idorder_idvaluewrittennote
no rows yet

Consumer group order-processor

stablegeneration 1lag 0

RangeAssignor (eager)commit after each recordmax.poll.records 5

  • processor-1runningprocessed 0
    orders-0
Partitions of group order-processor: owner, committed offset and lag
PartitionOwnerCommittedLag
orders-0 processor-1none0

Partition logs

simulated records
  • orders-0leader b1ISR 1,2,3HW 0end 0

    empty

    order-processor: committed none, lag 0

  • payments-0leader b1ISR 1,2,3HW 0end 0

    empty

    payment-gateway: committed none, lag 0

  • audit-0leader b1ISR 1,2,3HW 0end 0

    empty

    audit-store: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders (1 partitions, RF 3), payments (1 partitions, RF 3), audit (1 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.