Kafka & events · Production incident
The lag climbs while the consumers work
A consumer group's lag keeps growing although its consumers are alive. How do you find out what is stuck, and what do you change?
Thursday 10:05 · notification-serviceIllustrative scenario, not a real incident
Lag for group notifier on order-events is 410,000 records and rising
The notification service reads order-events with eight consumers and sends one email per event through a mail provider. It commits its offsets after each batch.
The group's lag was near zero until 09:31 and has grown in a straight line since. All eight pods are running and pass their health checks. One pod was restarted by hand at 09:50 and has read nothing since.
What the dashboards and logs show
- Lag, group notifier
- near 0 at 09:31, 410,000 at 10:05
- Emails sent
- about 200 a second, the usual rate
- Pods
- 8 of 8 running and ready, one restarted at 09:50
- Topic order-events
- every partition has a leader, produce rate as usual
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Emails are going out, but none of that progress is being recorded. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.