Skip to content
Practice

SEV-1 simulationDatabases · Real outage · Reconstructed data · 15–20 minutes

Incident #007 — Five backups, zero restores

Customer impact

The site is down. About 300 GB of the primary database's data directory has just been deleted.

SignalCurrent value
Sitedown
db1 (primary)≈300 GB deleted
db2 (secondary)out of sync
pg_dump backupsno alerts received
Cloud disk snapshotsNFS servers only
LVM snapshot17:20, for staging

Your task

It is 23:31 and you have just joined. Work out which copy of the database you can trust, using the service map, metrics, logs, traces and recent changes. Choose how to recover, then name the cause and the permanent fix.

23:31Impact continuing · +0 minabout 0 users unable to work lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

The primary's data is gone and the site is down. What do you restore from?