Observability · On-call judgement
Pages for the CPU, silence for the customers
How do you design alerting that pages on customer impact, not CPU?
Monday 10:00 · on-call reviewIllustrative scenario, not a real incident
On-call was paged at 01:24 for nothing, and nobody was paged when 5% of orders began to fail
The order API has an SLO: 99.9% of requests succeed over 30 days. Its only paging alert is on CPU, written years ago.
Last night the alert paged for a report job that harmed nobody. An hour later a deploy made one request in twenty fail. The CPU stayed flat, the pager stayed silent, and support found out from customers.
What the dashboards and logs show
- SLO
- 99.9% of requests succeed, over 30 days
- Paging alert
- CPU above 85% for 5 minutes
- Pages last night
- one, at 01:24
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
The new alerting will take a week to design and review. What do you do first, for tonight?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.