Skip to content
All outages

Incident · Hard · Kafka / streaming · Databases

Highly available, in one building

  • SEV1
  • ~41 HOURS
  • CONTROL PLANE
90s

The situation

One of your three core data centres loses utility power. The batteries, rated for ten minutes, last about four, and the generators do not take over. Your edge network keeps serving traffic. But the dashboard, the API and analytics all go down, even though those services are designed to run across all three sites.

The evidence

Edge traffic served vs dashboard/API availabilityillustrative
  • edge
  • dashboard/API
Logs and signalsReconstructed
11:43site A: power lost · batteries drained in ~4 min
11:44service 'config-api' (3-site HA): unavailable
11:44 ↳ needs message queue cluster → site A only
11:44 ↳ needs analytics database → site A only
12:30disaster-recovery failover: never tested end to end

Your call

Why did multi-site services fail?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem