Kafka & events · Production incident
Back after nine days, and nothing to catch up on
A Kafka consumer returns after nine days offline and reports no lag, yet its work for those days is missing. How do you recover it?
Friday 13 November 10:15 · loyalty-pointsIllustrative scenario, not a real incident
Support: 640 tickets about loyalty points missing for purchases made between 3 and 12 November
The loyalty service consumes the purchases topic and awards points for each purchase. It was switched off on 3 November for a database migration that overran, and was started again yesterday at 08:00.
It started without errors and its lag was at zero within seconds. Points for purchases made since yesterday morning are fine. Points for the nine days before that are missing.
What the dashboards and logs show
- Service stopped
- 3 November 08:00 to 12 November 08:00
- Lag after the start
- 0 on all 12 partitions within 5 seconds
- Errors at start-up
- none
- Points awarded, 3 to 12 November
- 0, against about 90,000 a day normally
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Customers are asking where their points are. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.