Incident · Medium · Databases · Infra
Healthy storage, refused requests
- SEV1
- ONE REGION
- VM DISKS
90s
The situation
Across one cloud region, virtual machines lose access to their disks. Storage reports that it is healthy and is simply refusing the requests. A routine host change went out region-wide over about an hour. Oddly, a globally distributed database that keeps management state there is degrading too.
The evidence
Logs and signalsReconstructed
t-70mhost change needs storage allow-list updatet-65mgenerator: some source files unreachable … continuingt-60mallow-list published to region (all zones, ~1h)t-0 storage: request rejected: address not in allow-listt+5m disks: healthy · replicas: healthy
Your call
Why are the disks unreachable?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.