Kubernetes & deploys · Production incident
The new pods will not stay up
After a deploy the new pods are in CrashLoopBackOff. How do you find out why the container keeps exiting?
Wednesday 14:20 · invoice-api, 8 replicasIllustrative scenario, not a real incident
invoice-api: 4 pods in CrashLoopBackOff since the 14:02 deploy
Version 4.12 of the invoice API was deployed at 14:02 to a Deployment of 8 replicas with the default rolling update settings.
Eighteen minutes later kubectl get pods lists 6 pods of the old version and 4 of the new. The 4 new ones show CrashLoopBackOff, and their restart counts keep rising.
What the dashboards and logs show
- Old version
- 6 pods, Running and Ready
- New version
- 4 pods, CrashLoopBackOff, 6 restarts each
- p99 latency
- 260 ms, up from 180 ms before 14:02
- Image pull
- succeeded on every node
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Six old pods are serving and four new ones are crashing. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.