BuildBreak lab 6 of 15
Consumer crash before offset commit
Kill a consumer between processing and commit, read the committed offset to predict the replay, size the replay window, then make the replay harmless.
Read first: Offset management and commits · The consumer and its poll loop · At-most-once, at-least-once and exactly-once · Duplicates: producer idempotence vs consumer idempotency
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- order-confirmed: 1 partition, RF 3
- Producers
- checkout
- Consumer groups
- notification-service (1)
- Guided run
- 7 stages: build → break → observe → fix → retest
1
Business scenario
The notification service consumes order-confirmed and sends each customer a confirmation email. It was written with the plain Java client and the settings from a tutorial: enable.auto.commit=true, committing every 5 seconds. Last night the node it ran on was drained and the pod was killed without warning. This morning support has a list of customers who received their confirmation twice, and nobody can say how many there are or why it was that many. The answer is in two numbers: the group's committed offset, and how far the consumer had really got.
2
Starting architecture
Drawing diagram…
The checkout service producer writes OrderConfirmed events to the order-confirmed topic. The notification service consumer polls the topic, first sends the email through the email provider, second records the send in the sent_emails table, and third, some time later, commits the offset to the group's committed offsets in Kafka.
flowchart LR
CO["Checkout service producer"] -->|"OrderConfirmed"| T["Topic order-confirmed"]
T -->|"poll"| NS["Notification service consumer"]
NS -->|"1 send the email"| M["Email provider"]
NS -->|"2 record the send"| DB[("sent_emails table")]
NS -->|"3 commit the offset, some time later"| OFF["Committed offsets of the group"]3
Expected behaviour
- Every confirmed order leads to exactly one email, whether or not the consumer is restarted in between.
- After a crash the consumer resumes at the group's committed offset. Anything it had processed beyond that offset is delivered again, and you can say in advance how many records that is.
- No order is skipped: the committed offset never moves past a record that has not been processed.
4 · Build → break → observe → fix → retest
Guided run
Send emails with auto-commit
Do this
- Turn on Checkout publishing (4 orders per second) and press play.
- Watch the Offsets panel: "processed up to" climbs record by record, while "committed" stands still and then jumps once every 5 seconds.
Look for
- "Processed, not committed" grows to about 20 and drops to 0 at each auto-commit. That number is what a crash at that moment would replay.
- The committed offset is the next offset to read, so "committed 40" means offsets 0 to 39 are done.
- The 5 seconds are Kafka's real default for
auto.commit.interval.ms; this value is not scaled down.
Not done yet. The simulator checks this for you:inserted("sent_emails") >= 20 && committed("notification-service", "order-confirmed", 0) > 0
Simulator
Simulation · not a real clusterControls for this stage
4 OrderConfirmed events per second.
Offsets of notification-service on order-confirmed
simulatedThe committed offset is stored in Kafka and is where a restart resumes. "Processed up to" lives only in the consumer's memory and dies with it.
- Processed, not committed (a crash now replays these)
- 0
- Committed, not processed (a crash now skips these)
- 0
Per partition
| partition | log end | committed (next to read) | processed up to | gap |
|---|---|---|---|---|
| order-confirmed-0 | 0 | 0 | nothing yet | none |
Simulated metrics
computed by the simulation- Committed offset (next to read)
- 0
- Processed up to (next offset)
- 0
- Replayed after a restart
- 0records
- Emails sent twice
- 0
- Replays absorbed by the idempotent handler
- 0
- Orders committed and never emailed
- 0
- Consumer crashes
- 0
- Lag
- 0records
Partition logs
simulated records- order-confirmed-0leader b1ISR 1,2,3HW 0end 0
empty
notification-service: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Consumer group notification-service
stablegeneration 1lag 0RangeAssignor (eager)auto-commitmax.poll.records 20
- notifier-1runningprocessed 0order-confirmed-0
| Partition | Owner | Committed | Lag |
|---|---|---|---|
| order-confirmed-0 | notifier-1 | none | 0 |
Database
simulated · available| id | event_id | value | written | note |
|---|---|---|---|---|
| no rows yet | ||||
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, order-confirmed (1 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.