BuildBreak lab 5 of 15
Duplicate message processing
A payment consumer charges a card, crashes before committing its offset, and charges again after the restart. Make the consumer idempotent and retest.
Read first: Offset management and commits · At-most-once, at-least-once and exactly-once · The idempotent consumer pattern
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- payment-requests: 3 partitions, RF 3
- Producers
- checkout
- Consumer groups
- payment-service (1)
- Guided run
- 6 stages: build → break → observe → fix → retest
1
Business scenario
The payment service consumes payment-requests. For each request it charges the customer's card and records the charge in its payments table, and then the offset is committed. On Friday evening the instance was killed by an out-of-memory error in the middle of a busy period and restarted by the orchestrator within seconds. On Monday a customer sends two bank statement lines for one order. The logs show one request, one record, one offset, and two charges.
2
Starting architecture
Drawing diagram…
The checkout service producer writes PaymentRequested events to the payment-requests topic. The payment service consumer polls the topic, first charges the card and inserts a row into the payments table, and second commits the offset to the group offsets. The two writes go to different systems.
flowchart LR
CO["Checkout service producer"] -->|"PaymentRequested"| T["Topic payment-requests"]
T -->|"poll"| PS["Payment service consumer"]
PS -->|"1 charge the card and insert a row"| DB[("payments table")]
PS -->|"2 commit offset"| OFF["Group offsets for payment-service"]3
Expected behaviour
- Every payment request leads to exactly one row in
paymentsand one charge on the card. - The consumer can crash at any point and be restarted without a customer being charged twice or not at all.
- A request that is delivered a second time is recognised and has no second effect.
4 · Build → break → observe → fix → retest
Guided run
Take payments
Do this
- Turn on Checkout publishing (2 payment requests per second) and press play.
- Watch the payments table fill and the committed offset follow the consumer in the partition logs.
Look for
- One row per request; "Double charges" is 0.
- The offset is committed after the charge: the commit mode is "after the batch is processed", Spring's default.
Not done yet. The simulator checks this for you:inserted("payments") >= 8 && duplicateRows("payments") == 0
Simulator
Simulation · not a real clusterControls for this stage
2 payment requests per second.
Database
simulated · available| id | payment_id | value | written | note |
|---|---|---|---|---|
| no rows yet | ||||
Simulated metrics
computed by the simulation- Payment requests
- 0
- Charges made
- 0
- Double charges
- 0
- Redeliveries
- 0
- Duplicate inserts rejected
- 0
- Requests committed and never charged
- 0
- Consumer crashes
- 0
- Lag
- 0records
Partition logs
simulated records- payment-requests-0leader b1ISR 1,2,3HW 0end 0
empty
payment-service: committed none, lag 0
- payment-requests-1leader b2ISR 2,3,1HW 0end 0
empty
payment-service: committed none, lag 0
- payment-requests-2leader b3ISR 3,1,2HW 0end 0
empty
payment-service: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Consumer group payment-service
stablegeneration 1lag 0RangeAssignor (eager)commit after batchmax.poll.records 10
- payment-1runningprocessed 0payment-requests-0payment-requests-1payment-requests-2
| Partition | Owner | Committed | Lag |
|---|---|---|---|
| payment-requests-0 | payment-1 | none | 0 |
| payment-requests-1 | payment-1 | none | 0 |
| payment-requests-2 | payment-1 | none | 0 |
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, payment-requests (3 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.