Skip to content
All outages

Incident · Hard · Payments · Databases

Fixed it. Broke it again.

  • SEV1
  • TWO WINDOWS
  • PAYMENTS API
90s

The situation

Your payments API degrades badly when a database cluster loses its primary and fails to elect a new one. The team restores service. Suspecting the recent database version, they roll the election protocol back to the old one. Hours later the API degrades a second time.

The evidence

API success rateillustrative
  • success %
Logs and signalsReconstructed
16:36shard primary failed
16:362 nodes stalled · health check: passing
16:37election: no primary elected
17:02recovered
20:10rollback: election protocol → previous version
21:14nodes failing · election protocol: previous · cluster config: current

Your call

What caused the second window?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem