Incident · Hard · DNS/HTTP · Infra
Nineteen sites, half the traffic
- SEV1
- 19 LOCATIONS
- ~50% OF REQUESTS
90s
The situation
Nineteen of your busiest data centres drop off the internet together. They are only about 4% of your locations but carry roughly half of all requests. You were rolling out a routing policy change that passed peer review and a dry run, in stages. These nineteen, which use a newer network design, were the final stage.
The evidence
Logs and signalsReconstructed
03:56policy change: early stages (older sites) → no impact06:27policy change: final stage (19 newer-design sites)06:27policy order: … REJECT-THE-REST · site-local prefixes06:2819 sites: prefixes withdrawn06:58reverts by several engineers overwrite each other
Your call
What went wrong?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.