Observability · Production incident
One pod at 100% CPU, seven peers at 30%
One pod of a service sits at 100% CPU while its peers are fine. What do you check, and in what order?
Tuesday 15:10 · search-api, 8 podsIllustrative scenario, not a real incident
search-api: pod search-api-7c9f at 100% of its CPU limit for 6 minutes
The search API is a Java 21 service on 8 pods, each with a CPU limit of 1. At 15:04 one pod went from 30% to 100% of its limit within a few seconds and has stayed there.
The other seven pods are at about 30%. Requests served by the busy pod have a p99 of 1.8 seconds, against 90 ms elsewhere. Overall traffic is flat.
What the dashboards and logs show
- CPU, search-api-7c9f
- 100% of its limit since 15:04
- CPU, the other 7 pods
- 28% to 33%
- p99 on the busy pod
- 1.8 s (90 ms on the others)
- Request rate, whole service
- flat at about 400 a second
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
One request in eight goes to a pod that answers in 1.8 seconds. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.