PersistentVolume issues
A pod cannot start because its volume cannot be bound, attached or mounted where the pod is.
$ kubectl get pods -o wideNAME READY STATUS RESTARTS AGE NODE db-5d8c7b9f6-h7t2m 1/1 Running 0 6d node-a db-6f9d7c5b8-n4k2x 0/1 ContainerCreating 0 4m12s node-b
$ kubectl describe pod db-6f9d7c5b8-n4k2xEvents: Type Reason Age From Message Normal Scheduled 4m12s default-scheduler Successfully assigned default/db-6f9d7c5b8-n4k2x to node-b Warning FailedAttachVolume 4m12s attachdetach-controller Multi-Attach error for volume "pvc-3f1c2a9e" Volume is already used by pod(s) db-5d8c7b9f6-h7t2m Warning FailedMount 2m9s kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data]: timed out waiting for the condition
$ kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data Bound pvc-3f1c2a9e 20Gi RWO standard 6d
Watch it happen
How PersistentVolume issues unfolds
Step 3 · The failure
The old pod still holds the disk
A rolling update starts the new pod before stopping the old one. The new pod lands on node B and asks for the disk, but it is still attached to node A. Every attempt is refused: Multi-Attach error. The new pod never becomes Ready, so the rollout never removes the old pod. Deadlock.
- attach retry
- refused
- waiting
- running
Step 3 · The failure
The old pod still holds the disk
A rolling update starts the new pod before stopping the old one. The new pod lands on node B and asks for the disk, but it is still attached to node A. Every attempt is refused: Multi-Attach error. The new pod never becomes Ready, so the rollout never removes the old pod. Deadlock.
- attach retry
- refused
- waiting
- running
Where it breaks: The disk and its rules are the constraint: a ReadWriteOnce disk attaches to one node at a time and lives in one zone. The attach/detach controller and the kubelet only report it, as Multi-Attach, FailedAttachVolume or FailedMount events.
In short
What PersistentVolume issues means
A pod with a volume needs three things to happen before its container starts. The PersistentVolumeClaim must be bound to a PersistentVolume, which may first have to be provisioned by a StorageClass. The disk must be attached to the node the pod was scheduled on. And the kubelet must mount it into the pod. Each stage fails differently. A claim that stays Pending has no matching volume or StorageClass, or is waiting on purpose for a pod to be scheduled. A Multi-Attach error means a ReadWriteOnce disk is still attached to another node, typically because the old pod of a rolling update is still running. A volume node affinity conflict means the disk is in one zone and the pod can only be placed in another. FailedMount is the kubelet giving up on waiting for any of these. The pod's events name the stage that is stuck.
In order
Diagnose in 60 seconds
- 1
kubectl get pvc- Look for
- STATUS: Bound or Pending. Note the ACCESS MODES and the STORAGECLASS.
- What it tells you
- Pending: the problem is binding or provisioning, before any node is involved. Bound: the volume exists, so look at attach and mount.
- 2
kubectl describe pvc <claim>- Look for
- The Events. “no persistent volumes available for this claim and no storage class is set”, a ProvisioningFailed message from the driver, or “waiting for first consumer to be created before binding”.
- What it tells you
- The first two are real failures. Waiting for first consumer is normal: the claim binds once a pod that uses it is scheduled.
- 3
kubectl describe pod <pod>- Look for
- The Events, in order: FailedScheduling with “volume node affinity conflict”, FailedAttachVolume with “Multi-Attach error”, or FailedMount.
- What it tells you
- Names the stage that is stuck: placement (zone), attach (another node holds the disk), or mount (on the node itself).
- 4
kubectl get volumeattachment- Look for
- Which NODE each volume is attached to, and whether ATTACHED is true.
- What it tells you
- Rules in a disk still held by another node, including a node that is NotReady or gone.
- 5
kubectl get pv <volume> -o jsonpath='{.spec.nodeAffinity}{"\n"}'- Look for
- The zone the volume is tied to. Compare with kubectl get nodes -L topology.kubernetes.io/zone.
- What it tells you
- Rules in a zone mismatch: the pod can only run where the disk lives.
- 6
kubectl get storageclass- Look for
- Whether one class is marked (default), its PROVISIONER and its VOLUMEBINDINGMODE.
- What it tells you
- No default class explains claims that never provision. Immediate binding explains disks created in a zone the pod cannot reach.
Most common first
Root causes
1A ReadWriteOnce volume is still attached to another node: a Deployment's rolling update, or a pod rescheduled while the old one lingers.
- Confirm
- FailedAttachVolume with “Multi-Attach error”, and kubectl get volumeattachment shows the other node.
- Fix now
- Remove the old pod so the disk is released. For a Deployment, switch to strategy Recreate.
- Prevent
- Use a StatefulSet, or Recreate, for anything that owns a single-writer disk.
2The disk is in one zone and the pod can only be scheduled in another.
- Confirm
- The pod is Pending with “volume node affinity conflict”, and the volume's node affinity names a zone with no usable node.
- Fix now
- Add capacity in the disk's zone, or restore the data to a new volume in a zone that has nodes.
- Prevent
- Use a StorageClass with volumeBindingMode WaitForFirstConsumer, so the disk is created where the pod lands.
3The claim cannot be provisioned: no default StorageClass, a class name that does not exist, or the CSI driver is not installed.
- Confirm
- The PVC stays Pending, and its events mention no storage class or a provisioning failure.
- Fix now
- Name an existing StorageClass in the claim, mark one as default, or install the driver.
- Prevent
- Check kubectl get storageclass on every new cluster before deploying stateful workloads.
4The node that held the disk died, and the volume has not been detached from it.
- Confirm
- The VolumeAttachment still points at a NotReady node, and the replacement pod waits in ContainerCreating.
- Fix now
- Confirm the machine is really off, then let the forced detach run or mark the node out-of-service.
- Prevent
- Plan for several minutes of failover for single-writer volumes, or replicate at the application level.
5The mount itself fails on the node: wrong filesystem, a missing Secret for the driver, or permissions on the volume.
- Confirm
- FailedMount with a specific message from the kubelet or driver, with the volume already attached.
- Fix now
- Fix what the message names: the Secret, the fsType, or the pod's fsGroup.
- Prevent
- Test a new StorageClass with a throwaway pod before pointing real workloads at it.
The tempting wrong fix
Delete the PVC and create it again
Before / after
The most common fix: do not roll a single-writer disk
kind: Deployment
spec:
replicas: 1
strategy:
type: RollingUpdate # starts the new pod first
template:
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: data # ReadWriteOncekind: Deployment
spec:
replicas: 1
strategy:
type: Recreate # stop the old pod, then start the new
template:
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: data # ReadWriteOnceReadWriteOnce allows several pods on the same node, so a rolling update can appear to work until the new pod lands on a different node. A StatefulSet is the better home for a pod that owns a disk.
In the real world
Depends on version and provider
Seen in a real outage
A zonal disk ties a workload to one place, however many nodes you run. At Cloudflare in 2023, services built to span three data centres went down when one lost power, because clusters they depended on existed only in that building.
Break it: Highly available, in one buildingYour turn
What would you do?
A single-replica Deployment uses a ReadWriteOnce volume. After a new image is rolled out, the new pod sits in ContainerCreating on another node and the old pod keeps running. What is happening?
Related errors
PersistentVolume issues questions
- Why is my pod stuck in ContainerCreating with a volume?
- The container cannot start until its volume is attached to the node and mounted. Run kubectl describe pod and read the events: a Multi-Attach error means another node still holds the disk, and FailedMount means the kubelet timed out waiting for it.
- What does Multi-Attach error for volume mean?
- The volume is ReadWriteOnce, so it can be attached to one node at a time, and it is still attached to a different node than the one the new pod was scheduled on. Releasing it there, usually by removing the old pod, lets the attach succeed.
- Why does my PVC stay Pending?
- Either nothing can satisfy it, because there is no matching volume, no StorageClass or a failing provisioner, or it is waiting on purpose. With WaitForFirstConsumer the claim binds only after a pod that uses it has been scheduled.
Sources: Kubernetes docs: Persistent volumes, access modes ↗ · Kubernetes docs: Storage classes, volume binding mode ↗ · Kubernetes docs: Node shutdowns, non-graceful shutdown and forced detach ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.