Practice
SEV-1 simulationDatabases · Real outage · Reconstructed data · 15–20 minutes
Incident #017 — The failover that never came
Customer impact
The API and dashboard are completely down. The primary database has lost its disks, and automatic failover has not happened.
| Signal | Current value |
|---|---|
| API and dashboard | down |
| Primary database | disks lost, powered off |
| Synchronous replica | healthy, fully up to date |
| Automatic failover | not promoting |
| Pacemaker error counts | cleared, no effect |
Your task
It is 15:18. The broken primary is off and the cluster manager's errors have been cleared, and still nothing has been promoted. Use the service map, metrics, logs, traces and configuration to decide what to do, then name the cause and the permanent fix.
15:18Impact continuing · +0 minabout 0 failed API requests (thousands) lost since you joined
Investigate · each first look costs 2 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time