Kafka & events · Production incident
Reserved twice after a restart
A Kafka consumer is killed after it processed a record and before it committed the offset. What happens next, and how do you make that safe?
Thursday 10:40 · reservation-serviceIllustrative scenario, not a real incident
Warehouse: three items show as sold out online while the shelves are full
The reservation service consumes order-placed. For each event it moves the ordered units from available to reserved in the stock database, and adds a row to its reservations table.
On Wednesday night the cluster's nodes were patched, and every consumer pod was replaced at least once. Since then some items have fewer units available than the warehouse counts.
What the dashboards and logs show
- Units held too many
- 212, across 37 items
- Orders overnight
- about 40 a second on each partition
- Pods replaced
- all 6, between 01:00 and 01:30
- Errors
- none in the service logs
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
The second half of the node patching is planned for tonight. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.