Skip to content
All outages

Incident · Hard · Databases · Infra

The storm of copies

  • SEV1
  • ONE ZONE → REGION
  • DAYS
90s

The situation

During a network upgrade, traffic for your block-storage cluster is shifted onto a low-capacity backup network by mistake. Storage nodes lose sight of their replicas. The mistake is corrected within minutes. Then volumes across the zone freeze anyway, and the control plane for the whole region starts timing out.

The evidence

Free cluster space vs volumes stuckillustrative
  • free space
  • stuck volumes
Logs and signalsReconstructed
00:47network change: traffic shifted to low-capacity network
00:52network corrected
00:53nodes: replica lost → searching for space to re-mirror
01:10cluster: no free space · nodes keep searching, no backoff
02:40regional API: thread pool exhausted waiting on this zone

Your call

Why did fixing the network not end it?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem