Incident · Hard · Databases · Infra
The storm of copies
- SEV1
- ONE ZONE → REGION
- DAYS
90s
The situation
During a network upgrade, traffic for your block-storage cluster is shifted onto a low-capacity backup network by mistake. Storage nodes lose sight of their replicas. The mistake is corrected within minutes. Then volumes across the zone freeze anyway, and the control plane for the whole region starts timing out.
The evidence
- free space
- stuck volumes
Logs and signalsReconstructed
00:47network change: traffic shifted to low-capacity network00:52network corrected00:53nodes: replica lost → searching for space to re-mirror01:10cluster: no free space · nodes keep searching, no backoff02:40regional API: thread pool exhausted waiting on this zone
Your call
Why did fixing the network not end it?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.