Kafka & events · Production incident
One record stops a partition
One Kafka record fails every time it is processed and the partition behind it stops. What should the consumer do with it?
Thursday 06:40 · shipment-serviceIllustrative scenario, not a real incident
Lag on order-events partition 3 is 48,000 and rising; partitions 0, 1, 2, 4 and 5 are at zero
The shipment service consumes order-events from six partitions and creates a shipment for each order. Since 02:10 no shipment has been created for any order on partition 3.
The consumer pods are up, their CPU is low, and nobody has deployed the shipment service this week. The warehouse opens at 08:00 and will have a sixth of its orders missing.
What the dashboards and logs show
- Lag, partition 3
- 48,000 and growing by about 180 a minute
- Lag, other partitions
- 0
- Consumer pods
- 3 of 3 running, no restarts, CPU 4%
- Shipments created since 02:10
- normal for five partitions, none for partition 3
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
The warehouse opens in 80 minutes and a sixth of its orders are missing. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.