Observability · Production incident
Slow requests that the application cannot explain
Request latency is high and nothing in the application accounts for it. Where do you look next?
Thursday 14:20 · orders-api, 10 podsIllustrative scenario, not a real incident
orders-api: p99 latency 620 ms for 20 minutes (objective 300 ms)
The orders API is a Java service on 10 pods. Its p99 has crept up over nine days and crossed the alert line today. The median is 45 ms and the error rate has not moved.
The team has already been through the application. The database answers in 12 ms, the one downstream call in 30 ms, and garbage collection pauses stay under 10 ms.
What the dashboards and logs show
- p99 latency
- 620 ms (140 ms nine days ago)
- Median latency
- 45 ms
- Database p99
- 12 ms
- Downstream call p99
- 30 ms
- Collector pauses
- under 10 ms
- Error rate
- 0.1%, unchanged
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
One request in a hundred takes over 600 ms, and the cause is unknown. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.