Practice
SEV-1 simulationDatabases · Real outage · Reconstructed data · 15–20 minutes
Incident #007 — Five backups, zero restores
Customer impact
The site is down. About 300 GB of the primary database's data directory has just been deleted.
| Signal | Current value |
|---|---|
| Site | down |
| db1 (primary) | ≈300 GB deleted |
| db2 (secondary) | out of sync |
| pg_dump backups | no alerts received |
| Cloud disk snapshots | NFS servers only |
| LVM snapshot | 17:20, for staging |
Your task
It is 23:31 and you have just joined. Work out which copy of the database you can trust, using the service map, metrics, logs, traces and recent changes. Choose how to recover, then name the cause and the permanent fix.
23:31Impact continuing · +0 minabout 0 users unable to work lost since you joined
Investigate · each first look costs 2 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time