Skip to content

BuildBreak lab 13 of 15

Database unavailable while consuming Kafka messages

The consumer's database goes down for a while. Try three common reactions, see what each one costs, then pause the listener, keep the offsets, and resume without losing a record.

Advanced35 min

Read first: Error handling, retries and dead letters in Spring · Poison messages and error handling · The idempotent consumer pattern · Kafka transactions and your database

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
orders-paid: 3 partitions, RF 3
Producers
order-service
Consumer groups
shipping-service (3)
Guided run
9 stages: build → break → observe → fix → retest

1

Business scenario

The shipping service consumes orders-paid and inserts one row per paid order into its shipments table; the warehouse works from that table. On Tuesday the database had a 90 second failover. Kafka and the shipping service stayed up the whole time and consumer lag never rose. On Wednesday the warehouse reports several hundred paid orders that were never shipped. The listener had caught the database exception, logged it, and returned normally.

2

Starting architecture

Drawing diagram…

The order service producer writes OrderPaid events to the topic orders-paid, which has three partitions. Three instances of the shipping service, shipping-1 to shipping-3, each poll one partition, insert a row into the shipments table, and then commit their offset to the group offsets of shipping-service.

flowchart LR
  OS["Order service producer"] -->|"OrderPaid"| T["Topic orders-paid, 3 partitions"]
  T -->|"poll"| S1["shipping-1"]
  T -->|"poll"| S2["shipping-2"]
  T -->|"poll"| S3["shipping-3"]
  S1 -->|"insert row"| DB[("shipments table")]
  S2 -->|"insert row"| DB
  S3 -->|"insert row"| DB
  S1 -->|"commit offset"| OFF["Group offsets for shipping-service"]
  S2 -->|"commit offset"| OFF
  S3 -->|"commit offset"| OFF
Each instance writes to the database first and commits its offset to Kafka second. The database can be down while Kafka is healthy.

3

Expected behaviour

  • Every paid order gets exactly one row in shipments, including orders paid while the database was down.
  • While the database is down the consumers wait: no offset moves past a record that was not written.
  • When the database is back the backlog drains by itself, with no rebalance and nothing to replay by hand.
  • Records that are broken in themselves (a poison message) still go to the dead letter topic; a database outage is not that.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 9

Ship paid orders

Do this

  1. Turn on Order service publishing (6 paid orders per second) and press play.
  2. Watch the shipments table fill and the lag stay near zero.

Look for

  • "Shipment rows written" follows "Orders paid".
  • The listener catches any exception from the write, logs it and returns normally. This is the code that was in production.

Not done yet. The simulator checks this for you:inserted("shipments") >= 20 && lag("shipping-service") < 10

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

6 paid orders per second.

Simulated metrics

computed by the simulation
Orders paid
0
Shipment rows written
0
Orders committed past with no shipment row
0
Lag
0records
Failed writes
0
Skipped
0
Dead-lettered
0
Removed for not polling
0
Rebalances
0
Failed commits
0
Duplicate shipment rows
0
Duplicate inserts rejected
0
Containers paused now
0
Container resumes
0

Listener containers (simulated)

simulated

When a database write fails: the group's error policy decides.

Database
available
Paused containers
0
Database probes
0
Retries inside the listener
0

Instances

instancestateholdingsince last poll()
shipping-1consuming0.0 s
shipping-2consuming0.0 s
shipping-3consuming0.0 s

Database

simulated · available
shipments0 rowsno unique constraint on event_id
Rows of table shipments: flagged rows first, then the newest
idevent_idvaluewrittennote
no rows yet

Consumer group shipping-service

stablegeneration 1lag 0

RangeAssignor (eager)commit after batchmax.poll.records 10

  • shipping-1runningprocessed 0
    orders-paid-0
  • shipping-2runningprocessed 0
    orders-paid-1
  • shipping-3runningprocessed 0
    orders-paid-2
Partitions of group shipping-service: owner, committed offset and lag
PartitionOwnerCommittedLag
orders-paid-0 shipping-1none0
orders-paid-1 shipping-2none0
orders-paid-2 shipping-3none0

Partition logs

simulated records
  • orders-paid-0leader b1ISR 1,2,3HW 0end 0

    empty

    shipping-service: committed none, lag 0

  • orders-paid-1leader b2ISR 2,3,1HW 0end 0

    empty

    shipping-service: committed none, lag 0

  • orders-paid-2leader b3ISR 3,1,2HW 0end 0

    empty

    shipping-service: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders-paid (3 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.