Skip to content
All outages

Editor's note: Based on press reporting, not a full postmortem.

Incident · Medium · Payments · Infra

Half-broken is worse than broken

  • SEV1
  • ~10 HOURS
  • CARD PAYMENTS
90s

The situation

Card payments across a continent start failing on a Friday afternoon. Not all of them: the failure rate swings between roughly 7% and 35%. You have two data centres and a backup network switch in each, built so that any one failure is invisible to shoppers.

The evidence

Transaction failure rateillustrative
  • failed %
Logs and signalsReconstructed
14:35primary core switch: partial fault (still responding)
14:35backup switch: waiting for primary to be declared dead
14:40transactions: intermittent failures
hourssite failover: delayed

Your call

Why didn't redundancy save you?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Press reporting