Kubernetes & deploys · Production incident
Running for an hour, Ready for none of it
The new pods are Running, never become Ready, and are restarted again and again. What do you change so that they start and take traffic?
Tuesday 11:40 · catalogue-search, 4 replicasIllustrative scenario, not a real incident
catalogue-search: rollout of 3.4 has made no progress for 40 minutes
Version 3.4 of the catalogue search service went out at 11:00. It is the first version that builds its search index in memory when it starts.
Since then the two new pods show Running with 0 of 1 containers Ready for about a minute at a time, and are then restarted. The rollout has not moved.
What the dashboards and logs show
- New pods
- 2, Running, 0/1 Ready, 9 restarts each
- Old pods
- 3, Running and Ready
- Search errors
- none
- Memory and CPU of new pods
- well under their limits
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Three old pods are serving and the rollout is stuck. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.