Skip to content
All outages

Incident · Medium · Infra

06:00 UTC, everywhere

  • SEV1
  • 06:00 UTC
  • 5 REGIONS, 3 CLOUDS
90s

The situation

At 06:00 UTC, tens of thousands of your Kubernetes nodes lose network connectivity. It happens in five regions across different cloud providers that share no network links. You replace a few nodes and the new ones work. So you start replacing them all, and hit the providers' rate limits.

The evidence

Nodes without connectivity, by regionillustrative
  • region A
  • region B
  • region C
Logs and signalsReconstructed
06:00apt: automatic security update → systemd patched
06:00systemd-networkd restarted → container routes removed
06:01node: pods unreachable
06:30new node booted → healthy
07:10mass node replacement → cloud API rate limited

Your call

What ties the regions together?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem