BuildBreak lab 13 of 15
Database unavailable while consuming Kafka messages
The consumer's database goes down for a while. Try three common reactions, see what each one costs, then pause the listener, keep the offsets, and resume without losing a record.
Read first: Error handling, retries and dead letters in Spring · Poison messages and error handling · The idempotent consumer pattern · Kafka transactions and your database
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- orders-paid: 3 partitions, RF 3
- Producers
- order-service
- Consumer groups
- shipping-service (3)
- Guided run
- 9 stages: build → break → observe → fix → retest
1
Business scenario
The shipping service consumes orders-paid and inserts one row per paid order into its shipments table; the warehouse works from that table. On Tuesday the database had a 90 second failover. Kafka and the shipping service stayed up the whole time and consumer lag never rose. On Wednesday the warehouse reports several hundred paid orders that were never shipped. The listener had caught the database exception, logged it, and returned normally.
2
Starting architecture
Drawing diagram…
The order service producer writes OrderPaid events to the topic orders-paid, which has three partitions. Three instances of the shipping service, shipping-1 to shipping-3, each poll one partition, insert a row into the shipments table, and then commit their offset to the group offsets of shipping-service.
flowchart LR
OS["Order service producer"] -->|"OrderPaid"| T["Topic orders-paid, 3 partitions"]
T -->|"poll"| S1["shipping-1"]
T -->|"poll"| S2["shipping-2"]
T -->|"poll"| S3["shipping-3"]
S1 -->|"insert row"| DB[("shipments table")]
S2 -->|"insert row"| DB
S3 -->|"insert row"| DB
S1 -->|"commit offset"| OFF["Group offsets for shipping-service"]
S2 -->|"commit offset"| OFF
S3 -->|"commit offset"| OFF3
Expected behaviour
- Every paid order gets exactly one row in
shipments, including orders paid while the database was down. - While the database is down the consumers wait: no offset moves past a record that was not written.
- When the database is back the backlog drains by itself, with no rebalance and nothing to replay by hand.
- Records that are broken in themselves (a poison message) still go to the dead letter topic; a database outage is not that.
4 · Build → break → observe → fix → retest
Guided run
Ship paid orders
Do this
- Turn on Order service publishing (6 paid orders per second) and press play.
- Watch the
shipmentstable fill and the lag stay near zero.
Look for
- "Shipment rows written" follows "Orders paid".
- The listener catches any exception from the write, logs it and returns normally. This is the code that was in production.
Not done yet. The simulator checks this for you:inserted("shipments") >= 20 && lag("shipping-service") < 10
Simulator
Simulation · not a real clusterControls for this stage
6 paid orders per second.
Simulated metrics
computed by the simulation- Orders paid
- 0
- Shipment rows written
- 0
- Orders committed past with no shipment row
- 0
- Lag
- 0records
- Failed writes
- 0
- Skipped
- 0
- Dead-lettered
- 0
- Removed for not polling
- 0
- Rebalances
- 0
- Failed commits
- 0
- Duplicate shipment rows
- 0
- Duplicate inserts rejected
- 0
- Containers paused now
- 0
- Container resumes
- 0
Listener containers (simulated)
simulatedWhen a database write fails: the group's error policy decides.
- Database
- available
- Paused containers
- 0
- Database probes
- 0
- Retries inside the listener
- 0
Instances
| instance | state | holding | since last poll() |
|---|---|---|---|
| shipping-1 | consuming | 0.0 s | |
| shipping-2 | consuming | 0.0 s | |
| shipping-3 | consuming | 0.0 s |
Database
simulated · available| id | event_id | value | written | note |
|---|---|---|---|---|
| no rows yet | ||||
Consumer group shipping-service
stablegeneration 1lag 0RangeAssignor (eager)commit after batchmax.poll.records 10
- shipping-1runningprocessed 0orders-paid-0
- shipping-2runningprocessed 0orders-paid-1
- shipping-3runningprocessed 0orders-paid-2
| Partition | Owner | Committed | Lag |
|---|---|---|---|
| orders-paid-0 | shipping-1 | none | 0 |
| orders-paid-1 | shipping-2 | none | 0 |
| orders-paid-2 | shipping-3 | none | 0 |
Partition logs
simulated records- orders-paid-0leader b1ISR 1,2,3HW 0end 0
empty
shipping-service: committed none, lag 0
- orders-paid-1leader b2ISR 2,3,1HW 0end 0
empty
shipping-service: committed none, lag 0
- orders-paid-2leader b3ISR 3,1,2HW 0end 0
empty
shipping-service: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders-paid (3 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.