Incident · Hard · Payments · Databases
Fixed it. Broke it again.
- SEV1
- TWO WINDOWS
- PAYMENTS API
90s
The situation
Your payments API degrades badly when a database cluster loses its primary and fails to elect a new one. The team restores service. Suspecting the recent database version, they roll the election protocol back to the old one. Hours later the API degrades a second time.
The evidence
- success %
Logs and signalsReconstructed
16:36shard primary failed16:362 nodes stalled · health check: passing16:37election: no primary elected17:02recovered20:10rollback: election protocol → previous version21:14nodes failing · election protocol: previous · cluster config: current
Your call
What caused the second window?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.