Skip to content
All outages

Incident · Hard · Infra · DNS/HTTP

Fail-static ran out

  • SEV1
  • SEVERAL US REGIONS
  • HOURS
90s

The situation

Scheduled maintenance begins in one location. For a few minutes nothing looks wrong. Then network capacity collapses across several regions as routes are withdrawn. The debugging tools you would use run over the same congested network and barely respond.

The evidence

Control-plane jobs running vs network capacityillustrative
  • control jobs
  • capacity
Logs and signalsReconstructed
11:45maintenance event: one location
11:46automation: descheduling network control jobs in multiple locations
11:46data plane: running on last known state (fail-static)
11:50fail-static grace period expired → BGP routes withdrawn
12:10debug tools: degraded by congestion

Your call

What caused the outage?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem