Observability · Production incident
The outage nobody was paged for
The monitoring system is down during an outage. How do you design a fallback?
Saturday 03:10 · production clusterIllustrative scenario, not a real incident
Checkout has been failing for 25 minutes, and the first report came from a customer, not from the pager
A bad node image rolled through the production cluster. Product pods are crash-looping, and the pods that run metrics, alerting and dashboards are not running either.
No alert fired, since the system that fires them is down. The dashboards do not load: their login page returns an error.
What the dashboards and logs show
- Pager
- silent since 02:44
- Metrics and alerting
- pods pending
- Dashboards
- login fails
- First report
- a customer email at 03:09
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Checkout is failing and you have no graphs of your own. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.