Skip to content
Practice

SEV-1 simulationPlatform and OS · Real outage · Reconstructed data · 15–20 minutes

Incident #012 — 06:00 UTC, everywhere

Customer impact

The web app and APIs are unreachable, monitors are not alerting, and tens of thousands of nodes are offline across five regions.

SignalCurrent value
Web app and APIsunreachable
Nodes offlinetens of thousands
Regions affected5, same minute
Monitorsnot alerting
Links between regionsnone

Your task

You are an incident commander. Work out what took nodes offline in five unconnected regions at once, using the service map, metrics, logs, traces and recent changes. Then choose how to recover, and name the cause and the permanent fix.

06:35Impact continuing · +0 minabout 0 thousand customers without monitoring lost since you joined

Investigate · each first look costs 3 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

The platform is down in five regions. How do you start recovering?