Skip to content
All outages

Incident · Hard · Payments

The back-out nobody tested

  • SEV0
  • 6.5M CUSTOMERS
  • WEEKS OF DISRUPTION
90s

The situation

You run the overnight batch for a group of banks. The scheduler was upgraded from v1 to v2 at the weekend, then patched to v2A. Tonight's batch runs slowly, so the team backs the upgrade out to v1. The next morning, jobs have vanished from the queues. Millions of customers see missing or duplicated transactions.

The evidence

Batch jobs outstandingillustrative · each day's work lands on top of the last
  • backlog
Logs and signalsReconstructed
Sun scheduler upgraded v1 → v2 → v2A
Tue 19:00batch slow · decision: back out to v1
-- back-out test on record: v2 → v1 only
Tue 23:00v1 started on v2A's queue data → queued jobs missing
Wed manual reruns overlap with today's batch · some payments applied twice

Your call

What turned a slow batch into a crisis?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Regulator