Practice
SEV-1 simulationPlatform and OS · Real outage · Reconstructed data · 15–20 minutes
Incident #012 — 06:00 UTC, everywhere
Customer impact
The web app and APIs are unreachable, monitors are not alerting, and tens of thousands of nodes are offline across five regions.
| Signal | Current value |
|---|---|
| Web app and APIs | unreachable |
| Nodes offline | tens of thousands |
| Regions affected | 5, same minute |
| Monitors | not alerting |
| Links between regions | none |
Your task
You are an incident commander. Work out what took nodes offline in five unconnected regions at once, using the service map, metrics, logs, traces and recent changes. Then choose how to recover, and name the cause and the permanent fix.
06:35Impact continuing · +0 minabout 0 thousand customers without monitoring lost since you joined
Investigate · each first look costs 3 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time