Kafka & events · Production incident
Busy consumers, no progress
A consumer group keeps rebalancing and its throughput has collapsed. How do you find out why and make the group stable?
Wednesday 15:10 · search-indexerIllustrative scenario, not a real incident
Lag on listing-updated has passed 2 million records, and throughput is under a tenth of normal
The search indexer reads listing-updated with 12 consumers on 24 partitions. For each record it calls the pricing service, then writes the listing to the search index.
Since about 13:40 the lag has climbed without a pause. The index shows the same listings being written again and again, and no pod has restarted.
What the dashboards and logs show
- Throughput
- about 90 records a second, down from 1,200
- Lag
- 2.1 million records and rising
- Pods
- 12 of 12 ready, no restarts, CPU near 15%
- Index writes
- about 7 in 10 repeat an earlier write
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Search results are getting staler by the minute. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.