Incident · Hard · Payments
The back-out nobody tested
- SEV0
- 6.5M CUSTOMERS
- WEEKS OF DISRUPTION
90s
The situation
You run the overnight batch for a group of banks. The scheduler was upgraded from v1 to v2 at the weekend, then patched to v2A. Tonight's batch runs slowly, so the team backs the upgrade out to v1. The next morning, jobs have vanished from the queues. Millions of customers see missing or duplicated transactions.
The evidence
- backlog
Logs and signalsReconstructed
Sun scheduler upgraded v1 → v2 → v2ATue 19:00batch slow · decision: back out to v1-- back-out test on record: v2 → v1 onlyTue 23:00v1 started on v2A's queue data → queued jobs missingWed manual reruns overlap with today's batch · some payments applied twice
Your call
What turned a slow batch into a crisis?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.