Incident · Easy · Databases
Five backups, zero restores
- SEV1
- LATE NIGHT
- PRIMARY DB WIPED
90s
The situation
A replica is lagging and a tired engineer decides to rebuild it. They clear its data directory, then realise the terminal was connected to db1, the primary, not db2. Most of the production database is gone. No problem, someone says: restore last night's backup.
The evidence
Logs and signalsReconstructed
23:27db1 (primary) $ rm -rf data directory23:30nightly dump in object storage: bucket empty23:31dump tool: version mismatch, has been failing silently23:32failure alert emails: rejected by mail policy, never delivered23:40disk snapshot from 6h ago, taken by hand: exists
Your call
What is the real lesson here?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.