Kafka & events · Production incident
More consumers, same lag
One partition of a Kafka topic lags while the others are idle, and adding partitions and consumers did not help. What do you do?
Tuesday 11:30 · price-syncIllustrative scenario, not a real incident
Lag on listing-events is 4.1 million records and still rising, a day after capacity was doubled
The price-sync service consumes listing-events and applies each price or stock change to the public catalogue. Shoppers are seeing prices that are more than an hour old.
Yesterday, during a similar build-up, the team raised the topic from 12 partitions to 24 and the consumers from 12 to 24. This morning the lag is growing as fast as it did before that change.
What the dashboards and logs show
- Total lag
- 4.1 million records, growing by about 600 a second
- Produce rate
- 2,400 a second since 09:35, against about 1,400 on a normal morning
- Consumers
- 24, all in the group, average CPU 9%
- Errors
- none from producers or consumers
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Other merchants on the same partition are stuck behind the backlog. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.