Kafka & events · Production incident
Every deploy costs twenty minutes of lag
Consumer lag jumps after every deploy of a consumer service and takes twenty minutes to recover. How do you make it small?
Wednesday 14:30 · tracking-serviceIllustrative scenario, not a real incident
Lag on shipment-events passes 300,000 during every deploy of the tracking service
The tracking service reads shipment-events with 12 consumers on 48 partitions and updates the delivery status that customers see. It handles 2,400 records a second with room to spare.
Every deploy restarts the pods one at a time and takes about twelve minutes. Each time, the lag climbs throughout and needs another ten minutes to clear, so customers see statuses that are minutes old. The team deploys three times a day.
What the dashboards and logs show
- Lag
- under 1,000 normally, about 380,000 at the end of a deploy
- Deploy
- 12 pods, one at a time, each back in about 40 seconds
- Capacity
- 3,000 records a second with all 12 reading, 2,400 arriving
- Errors
- none, and no pod fails its health check
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
A deploy is halfway through, the lag is 190,000 and customers are seeing old statuses. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.