Kafka & events · Concept deep-dive
Once, at least once, or not at all
What delivery guarantees does Kafka give, where do they stop, and how do you get a result that happens once in a database?
Concept deep-dive · Kafka delivery guaranteesIllustrative scenario, not a real incident
Design review: the loyalty points pipeline must not award points twice, or lose them
The points service consumes order-paid. For each record it adds a row to a points ledger in PostgreSQL and publishes points-awarded, which the notification service turns into a message to the customer.
Twice last quarter a restart during a deploy left customers with double points. A proposal on the table says: turn on Kafka's strongest guarantee and the problem goes away.
What the dashboards and logs show
- Input
- order-paid, keyed by order id
- Effects per record
- one ledger row in PostgreSQL, one points-awarded record
- Offsets
- committed after each batch has been handled
- Ledger table
- no unique constraint on the order id
5 questions on one topic, each at least as hard as the last. Answer each before the next appears. The last one plays out.
Question 1 of 5 · Mid · scored under "Hypothesis before change"
A consumer saves its offset, then starts on the batch, and crashes halfway. What happens to the rest of the batch?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.