Skip to content
Interview rounds

Kafka & events · Production incident

Every deploy costs twenty minutes of lag

Consumer lag jumps after every deploy of a consumer service and takes twenty minutes to recover. How do you make it small?

BackendMid12 min
Interview-style scenarioLast reviewed: Checked against official documentation

Wednesday 14:30 · tracking-serviceIllustrative scenario, not a real incident

Lag on shipment-events passes 300,000 during every deploy of the tracking service

The tracking service reads shipment-events with 12 consumers on 48 partitions and updates the delivery status that customers see. It handles 2,400 records a second with room to spare.

Every deploy restarts the pods one at a time and takes about twelve minutes. Each time, the lag climbs throughout and needs another ten minutes to clear, so customers see statuses that are minutes old. The team deploys three times a day.

What the dashboards and logs show

Lag
under 1,000 normally, about 380,000 at the end of a deploy
Deploy
12 pods, one at a time, each back in about 40 seconds
Capacity
3,000 records a second with all 12 reading, 2,400 arriving
Errors
none, and no pod fails its health check

Clarify · optional · scored under "Hypothesis before change"

Ask the interviewer

0 of 3 asked

Each question reveals a fact. What you ask before you act is part of your score.

First move · scored under "Mitigate before root cause"

A deploy is halfway through, the lag is 190,000 and customers are seeing old statuses. What do you do first?

Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.