Skip to content

DNS / HTTP · 3 of 8

Step 1 · The pattern

One address, many sites

Three sites all announce the same IP address. Internet routing delivers each user's packets to the nearest one. This is anycast, and it is how DNS providers and CDNs stay close to everyone.

  • request
  • answered

In short

Anycast / BGP withdrawal: how it works and how it breaks

With anycast, many sites around the world announce the same IP address using BGP, the protocol networks use to tell each other which addresses they can reach. Each user's traffic flows to whichever announcing site is closest. If a site becomes unhealthy it withdraws its announcement, routes update within seconds, and its users are carried to the next nearest site. That self-healing rests on one question: how does a site decide it is unhealthy? If the test is 'can I reach the central data centres', then losing the internal backbone fails the test at every site in the same instant. All of them withdraw. The address is announced nowhere and the service vanishes from the internet, though every edge machine is running. The defence is to fail static. When every site would withdraw together, suspect the check, keep announcing, and serve what you can.

How this comes up in interviews

"A service announces one IP address from many sites using anycast. Every edge server is running, yet the service vanished worldwide. How?"

Then they ask

  1. How should a site decide it is unhealthy, and what should it do when every site fails the check together?
  2. Engineers cannot log in to fix the routers during the outage. What was missing from the design?
  3. What happens to a site's users, and to the neighbouring sites, when it withdraws its route?

A strong answer covers

  • Many sites announce the same address with BGP, and each user's traffic flows to the closest one.
  • An unhealthy site withdraws its announcement, and its users are carried to the next nearest site within seconds.
  • If the health check tests a shared dependency, such as the backbone, it fails at every site in the same instant.
  • Fail static: when every site would withdraw together, suspect the check, keep announcing and serve what you can.
  • Recovery tools need a path that does not depend on what they repair: out-of-band access, tested regularly.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Official postmortem

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.