Incident · Hard · Kafka / streaming · Databases
Highly available, in one building
- SEV1
- ~41 HOURS
- CONTROL PLANE
90s
The situation
One of your three core data centres loses utility power. The batteries, rated for ten minutes, last about four, and the generators do not take over. Your edge network keeps serving traffic. But the dashboard, the API and analytics all go down, even though those services are designed to run across all three sites.
The evidence
- edge
- dashboard/API
Logs and signalsReconstructed
11:43site A: power lost · batteries drained in ~4 min11:44service 'config-api' (3-site HA): unavailable11:44 ↳ needs message queue cluster → site A only11:44 ↳ needs analytics database → site A only12:30disaster-recovery failover: never tested end to end
Your call
Why did multi-site services fail?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.