Skip to content
All outages

Incident · Medium · Infra · DNS/HTTP

One command, every router

  • SEV1
  • GLOBAL WAN
  • TWO WAVES
90s

The situation

An engineer is adding capacity at one site in Madrid, following a written procedure. They run a command on one router. Seconds later, routers across your global wide-area network start recalculating their paths and dropping packets. Thirty-three minutes later it happens again.

The evidence

Packet loss, global WANillustrative
  • loss %
Logs and signalsReconstructed
07:05madrid-rtr: purge command run (per procedure)
07:05all routers in routing domain: recomputing forwarding tables
07:06traffic-engineering automation: paused
07:38procedure step repeated → second wave
-- command behaves as local-only on the other vendor's routers

Your call

Why did one router affect all of them?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem