Kafka & events · Production incident
Closed accounts return after a rebuild
A service rebuilds its state from a compacted Kafka topic and finds customers who closed their accounts. How should a deletion be written?
Wednesday 14:30 · marketing-serviceIllustrative scenario, not a real incident
Complaints: 1,900 customers who closed their accounts received Wednesday's marketing email
The marketing service keeps every customer profile in memory. At start-up each instance builds that state by reading the customer-profiles topic from offset 0, then keeps it up to date from new records.
On Tuesday night every instance was restarted for a routine upgrade of the operating system. Before that, the instances had run for five months. Wednesday's email went to 1,900 people whose accounts are closed.
What the dashboards and logs show
- Closed accounts emailed
- 1,900, closed at various times over the last five months
- Customers in memory
- 2,101,900 after the restart; 2,100,000 before it
- Restart
- Tuesday 23:00, all instances; the same build as before
- Accounts database
- all 1,900 are marked closed
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Another marketing email is scheduled for 17:00. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.