Skip to content
All outages

Incident · Hard · DNS/HTTP · Infra

Nineteen sites, half the traffic

  • SEV1
  • 19 LOCATIONS
  • ~50% OF REQUESTS
90s

The situation

Nineteen of your busiest data centres drop off the internet together. They are only about 4% of your locations but carry roughly half of all requests. You were rolling out a routing policy change that passed peer review and a dry run, in stages. These nineteen, which use a newer network design, were the final stage.

The evidence

Logs and signalsReconstructed
03:56policy change: early stages (older sites) → no impact
06:27policy change: final stage (19 newer-design sites)
06:27policy order: … REJECT-THE-REST · site-local prefixes
06:2819 sites: prefixes withdrawn
06:58reverts by several engineers overwrite each other

Your call

What went wrong?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem