Skip to content
All outages

Incident · Hard · DNS/HTTP · Infra

Name servers alive, nobody home

  • SEV0
  • ~6 HOURS
  • EVERYTHING
90s

The situation

Every product you run disappears from the internet at once. Resolvers worldwide return SERVFAIL for your domains. Your name servers are powered on and healthy, but nobody can route to them. Internal tools and chat are failing too. Minutes earlier, a routine maintenance command ran on the backbone that links your data centres.

The evidence

Backbone links vs DNS routes announcedillustrative
  • backbone links up
  • DNS routes announced
Logs and signalsReconstructed
15:39backbone: capacity-assessment command issued
15:39audit tool: command not blocked
15:40backbone: all links down
15:40edge DNS: cannot reach data centres → withdrawing own BGP routes
15:41world: SERVFAIL
15:45remote access tools: cannot resolve

Your call

What is the root cause?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem