Incident · Medium · Infra
06:00 UTC, everywhere
- SEV1
- 06:00 UTC
- 5 REGIONS, 3 CLOUDS
90s
The situation
At 06:00 UTC, tens of thousands of your Kubernetes nodes lose network connectivity. It happens in five regions across different cloud providers that share no network links. You replace a few nodes and the new ones work. So you start replacing them all, and hit the providers' rate limits.
The evidence
- region A
- region B
- region C
Logs and signalsReconstructed
06:00apt: automatic security update → systemd patched06:00systemd-networkd restarted → container routes removed06:01node: pods unreachable06:30new node booted → healthy07:10mass node replacement → cloud API rate limited
Your call
What ties the regions together?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.