Incident · Medium · Databases
Right script, wrong IDs
- SEV1
- 775 CUSTOMERS
- UP TO 14 DAYS
90s
The situation
A maintenance script is meant to remove an old add-on from a list of customer sites. It runs without errors. Shortly after, 883 entire customer sites no longer exist. You have backups with a recovery point of minutes. Leadership asks how soon everyone will be back.
The evidence
- restored %
Logs and signalsReconstructed
07:38script input: list of IDs (site IDs, not add-on IDs)07:38delete API: accepts either ID type, no type check07:39883 sites permanently deleted (no soft delete)day 1restore tooling: one site at a time, by hand
Your call
Why will recovery be slow?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.