Skip to content
Practice

SEV-1 simulationDatabases · Real outage · Reconstructed data · 15–20 minutes

Incident #017 — The failover that never came

Customer impact

The API and dashboard are completely down. The primary database has lost its disks, and automatic failover has not happened.

SignalCurrent value
API and dashboarddown
Primary databasedisks lost, powered off
Synchronous replicahealthy, fully up to date
Automatic failovernot promoting
Pacemaker error countscleared, no effect

Your task

It is 15:18. The broken primary is off and the cluster manager's errors have been cleared, and still nothing has been promoted. Use the service map, metrics, logs, traces and configuration to decide what to do, then name the cause and the permanent fix.

15:18Impact continuing · +0 minabout 0 failed API requests (thousands) lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

The automation has not promoted anything. What do you do?