Kafka & events · Production incident
The refund that was paid twice
A Kafka consumer processes the same event twice after a restart. How do you make the second delivery harmless?
Tuesday 16:40 · refund-serviceIllustrative scenario, not a real incident
Finance: 14 refunds paid twice since this afternoon's deploy
The refund service consumes refund-requests. For each event it pays the customer through the bank and inserts a row into its refunds table.
At 14:05 a rolling deploy restarted the consumers. Since then finance has found refunds that were paid twice.
What the dashboards and logs show
- Double refunds
- 14, all since the 14:05 deploy
- refunds table
- two rows per affected refund id
- Deploy
- rolling restart; two pods were killed for stopping too slowly
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Another deploy is planned for this afternoon. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.