Kafka & events · Production incident
The confirmation emails that arrive late
Consumer lag on a Kafka topic is above its alert threshold. Is it a passing spike or a real problem, and what do you do?
Friday 19:30 · email-senderIllustrative scenario, not a real incident
Consumer lag on order-confirmations is above its threshold
The email service consumes order-confirmations and calls an email provider once for each record. A sale started at 19:00.
Customers now get their confirmation several minutes after paying, and the lag alert fired a few minutes ago. Nothing is failing: no errors, no restarts, no rebalances.
What the dashboards and logs show
- Lag
- about 10,800 records in total; the alert threshold is 1,000
- Topic
- order-confirmations, 3 partitions
- Consumer group
- 3 instances
- Consumer settings
- max.poll.records 500, max.poll.interval.ms 300000 (defaults)
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
The alert has just fired and support is asking what to tell customers. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.