Kubernetes & deploys · Production incident
The deploy succeeds, and customers see errors
Every rolling deploy reports success, and every one produces a burst of 5xx. What do you change so that a rollout drops no requests?
Thursday 15:10 · checkout-api, 12 replicasIllustrative scenario, not a real incident
checkout-api: 5xx rate 2.1% for four minutes, third time this week
The checkout API runs as a Deployment of 12 pods behind an ingress gateway. Each of this week's three releases finished with kubectl rollout status reporting success.
During each rollout the gateway logged about 300 responses with status 502 or 503. Between rollouts the count is zero, and the three releases changed unrelated code.
What the dashboards and logs show
- 5xx at the gateway
- about 300 per rollout, none between rollouts
- Rollout
- about four minutes, reported as successful each time
- Pod restarts
- 0 on every pod, old and new
- Application logs
- no exceptions during the rollouts
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
The fix needs a release of its own. What do you do about the errors until then?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.