Skip to content
All outages

Incident · Medium · Databases · Infra

Healthy storage, refused requests

  • SEV1
  • ONE REGION
  • VM DISKS
90s

The situation

Across one cloud region, virtual machines lose access to their disks. Storage reports that it is healthy and is simply refusing the requests. A routine host change went out region-wide over about an hour. Oddly, a globally distributed database that keeps management state there is degrading too.

The evidence

Logs and signalsReconstructed
t-70mhost change needs storage allow-list update
t-65mgenerator: some source files unreachable … continuing
t-60mallow-list published to region (all zones, ~1h)
t-0 storage: request rejected: address not in allow-list
t+5m disks: healthy · replicas: healthy

Your call

Why are the disks unreachable?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem