Incident · Medium · Payments · Infra
Half-broken is worse than broken
- SEV1
- ~10 HOURS
- CARD PAYMENTS
90s
The situation
Card payments across a continent start failing on a Friday afternoon. Not all of them: the failure rate swings between roughly 7% and 35%. You have two data centres and a backup network switch in each, built so that any one failure is invisible to shoppers.
The evidence
- failed %
Logs and signalsReconstructed
14:35primary core switch: partial fault (still responding)14:35backup switch: waiting for primary to be declared dead14:40transactions: intermittent failureshourssite failover: delayed
Your call
Why didn't redundancy save you?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.