Skip to content

BuildBreak lab 5 of 15

Duplicate message processing

A payment consumer charges a card, crashes before committing its offset, and charges again after the restart. Make the consumer idempotent and retest.

Intermediate25 min

Read first: Offset management and commits · At-most-once, at-least-once and exactly-once · The idempotent consumer pattern

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
payment-requests: 3 partitions, RF 3
Producers
checkout
Consumer groups
payment-service (1)
Guided run
6 stages: build → break → observe → fix → retest

1

Business scenario

The payment service consumes payment-requests. For each request it charges the customer's card and records the charge in its payments table, and then the offset is committed. On Friday evening the instance was killed by an out-of-memory error in the middle of a busy period and restarted by the orchestrator within seconds. On Monday a customer sends two bank statement lines for one order. The logs show one request, one record, one offset, and two charges.

2

Starting architecture

Drawing diagram…

The checkout service producer writes PaymentRequested events to the payment-requests topic. The payment service consumer polls the topic, first charges the card and inserts a row into the payments table, and second commits the offset to the group offsets. The two writes go to different systems.

flowchart LR
  CO["Checkout service producer"] -->|"PaymentRequested"| T["Topic payment-requests"]
  T -->|"poll"| PS["Payment service consumer"]
  PS -->|"1 charge the card and insert a row"| DB[("payments table")]
  PS -->|"2 commit offset"| OFF["Group offsets for payment-service"]
The charge and the offset commit are two separate writes to two separate systems, in that order.

3

Expected behaviour

  • Every payment request leads to exactly one row in payments and one charge on the card.
  • The consumer can crash at any point and be restarted without a customer being charged twice or not at all.
  • A request that is delivered a second time is recognised and has no second effect.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 6

Take payments

Do this

  1. Turn on Checkout publishing (2 payment requests per second) and press play.
  2. Watch the payments table fill and the committed offset follow the consumer in the partition logs.

Look for

  • One row per request; "Double charges" is 0.
  • The offset is committed after the charge: the commit mode is "after the batch is processed", Spring's default.

Not done yet. The simulator checks this for you:inserted("payments") >= 8 && duplicateRows("payments") == 0

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

2 payment requests per second.

Database

simulated · available
payments0 rowsno unique constraint on payment_id
Rows of table payments: flagged rows first, then the newest
idpayment_idvaluewrittennote
no rows yet

Simulated metrics

computed by the simulation
Payment requests
0
Charges made
0
Double charges
0
Redeliveries
0
Duplicate inserts rejected
0
Requests committed and never charged
0
Consumer crashes
0
Lag
0records

Partition logs

simulated records
  • payment-requests-0leader b1ISR 1,2,3HW 0end 0

    empty

    payment-service: committed none, lag 0

  • payment-requests-1leader b2ISR 2,3,1HW 0end 0

    empty

    payment-service: committed none, lag 0

  • payment-requests-2leader b3ISR 3,1,2HW 0end 0

    empty

    payment-service: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Consumer group payment-service

stablegeneration 1lag 0

RangeAssignor (eager)commit after batchmax.poll.records 10

  • payment-1runningprocessed 0
    payment-requests-0payment-requests-1payment-requests-2
Partitions of group payment-service: owner, committed offset and lag
PartitionOwnerCommittedLag
payment-requests-0 payment-1none0
payment-requests-1 payment-1none0
payment-requests-2 payment-1none0

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, payment-requests (3 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.