Skip to content
All outages

Incident · Medium · Databases

Right script, wrong IDs

  • SEV1
  • 775 CUSTOMERS
  • UP TO 14 DAYS
90s

The situation

A maintenance script is meant to remove an old add-on from a list of customer sites. It runs without errors. Shortly after, 883 entire customer sites no longer exist. You have backups with a recovery point of minutes. Leadership asks how soon everyone will be back.

The evidence

Sites restoredillustrative · days after deletion
  • restored %
Logs and signalsReconstructed
07:38script input: list of IDs (site IDs, not add-on IDs)
07:38delete API: accepts either ID type, no type check
07:39883 sites permanently deleted (no soft delete)
day 1restore tooling: one site at a time, by hand

Your call

Why will recovery be slow?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem