Skip to content
All outages

Incident · Easy · Databases

Five backups, zero restores

  • SEV1
  • LATE NIGHT
  • PRIMARY DB WIPED
90s

The situation

A replica is lagging and a tired engineer decides to rebuild it. They clear its data directory, then realise the terminal was connected to db1, the primary, not db2. Most of the production database is gone. No problem, someone says: restore last night's backup.

The evidence

Logs and signalsReconstructed
23:27db1 (primary) $ rm -rf data directory
23:30nightly dump in object storage: bucket empty
23:31dump tool: version mismatch, has been failing silently
23:32failure alert emails: rejected by mail policy, never delivered
23:40disk snapshot from 6h ago, taken by hand: exists

Your call

What is the real lesson here?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem