Skip to content

BuildBreak lab 6 of 15

Consumer crash before offset commit

Kill a consumer between processing and commit, read the committed offset to predict the replay, size the replay window, then make the replay harmless.

Intermediate25 min

Read first: Offset management and commits · The consumer and its poll loop · At-most-once, at-least-once and exactly-once · Duplicates: producer idempotence vs consumer idempotency

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
order-confirmed: 1 partition, RF 3
Producers
checkout
Consumer groups
notification-service (1)
Guided run
7 stages: build → break → observe → fix → retest

1

Business scenario

The notification service consumes order-confirmed and sends each customer a confirmation email. It was written with the plain Java client and the settings from a tutorial: enable.auto.commit=true, committing every 5 seconds. Last night the node it ran on was drained and the pod was killed without warning. This morning support has a list of customers who received their confirmation twice, and nobody can say how many there are or why it was that many. The answer is in two numbers: the group's committed offset, and how far the consumer had really got.

2

Starting architecture

Drawing diagram…

The checkout service producer writes OrderConfirmed events to the order-confirmed topic. The notification service consumer polls the topic, first sends the email through the email provider, second records the send in the sent_emails table, and third, some time later, commits the offset to the group's committed offsets in Kafka.

flowchart LR
  CO["Checkout service producer"] -->|"OrderConfirmed"| T["Topic order-confirmed"]
  T -->|"poll"| NS["Notification service consumer"]
  NS -->|"1 send the email"| M["Email provider"]
  NS -->|"2 record the send"| DB[("sent_emails table")]
  NS -->|"3 commit the offset, some time later"| OFF["Committed offsets of the group"]
The consumer's progress exists twice: in its memory (how far it has processed) and in Kafka (the committed offset). Only the second survives a crash.

3

Expected behaviour

  • Every confirmed order leads to exactly one email, whether or not the consumer is restarted in between.
  • After a crash the consumer resumes at the group's committed offset. Anything it had processed beyond that offset is delivered again, and you can say in advance how many records that is.
  • No order is skipped: the committed offset never moves past a record that has not been processed.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 7

Send emails with auto-commit

Do this

  1. Turn on Checkout publishing (4 orders per second) and press play.
  2. Watch the Offsets panel: "processed up to" climbs record by record, while "committed" stands still and then jumps once every 5 seconds.

Look for

  • "Processed, not committed" grows to about 20 and drops to 0 at each auto-commit. That number is what a crash at that moment would replay.
  • The committed offset is the next offset to read, so "committed 40" means offsets 0 to 39 are done.
  • The 5 seconds are Kafka's real default for auto.commit.interval.ms; this value is not scaled down.

Not done yet. The simulator checks this for you:inserted("sent_emails") >= 20 && committed("notification-service", "order-confirmed", 0) > 0

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

4 OrderConfirmed events per second.

Offsets of notification-service on order-confirmed

simulated

The committed offset is stored in Kafka and is where a restart resumes. "Processed up to" lives only in the consumer's memory and dies with it.

Processed, not committed (a crash now replays these)
0
Committed, not processed (a crash now skips these)
0

Per partition

partitionlog endcommitted (next to read)processed up togap
order-confirmed-000nothing yetnone

Simulated metrics

computed by the simulation
Committed offset (next to read)
0
Processed up to (next offset)
0
Replayed after a restart
0records
Emails sent twice
0
Replays absorbed by the idempotent handler
0
Orders committed and never emailed
0
Consumer crashes
0
Lag
0records

Partition logs

simulated records
  • order-confirmed-0leader b1ISR 1,2,3HW 0end 0

    empty

    notification-service: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Consumer group notification-service

stablegeneration 1lag 0

RangeAssignor (eager)auto-commitmax.poll.records 20

  • notifier-1runningprocessed 0
    order-confirmed-0
Partitions of group notification-service: owner, committed offset and lag
PartitionOwnerCommittedLag
order-confirmed-0 notifier-1none0

Database

simulated · available
sent_emails0 rowsno unique constraint on event_id
Rows of table sent_emails: flagged rows first, then the newest
idevent_idvaluewrittennote
no rows yet

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, order-confirmed (1 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.