Observability · Concept deep-dive
The average is fine, and users time out
Average latency looks healthy while users hit timeouts. Why, and what should you measure and alert on?
Concept deep-dive · latency and percentilesIllustrative scenario, not a real incident
Support: some customers say checkout hangs and then fails. The latency dashboard is green
The checkout service has one latency panel, the mean response time, and one alert on it: mean above 500 ms for five minutes. The mean is 119 ms and the alert has never fired.
The mobile app gives up on a request after 2 seconds. Since this afternoon a steady trickle of customers has been hitting that limit.
What the dashboards and logs show
- Mean latency
- 119 ms (60 ms this morning)
- Alert
- mean above 500 ms for 5 minutes: quiet
- Client timeout
- 2 seconds
- Requests
- about 1,000 a minute
5 questions on one topic, each at least as hard as the last. Answer each before the next appears. The last one plays out.
Question 1 of 5 · Mid · scored under "Hypothesis before change"
The mean is 119 ms and some users wait over 2 seconds. How can both be true?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.