Skip to content
Breaks at Cloud disk · CSI driverCan take the service downCommon

PersistentVolume issues

A pod cannot start because its volume cannot be bound, attached or mounted where the pod is.

What you'll seeIllustrative output
$ kubectl get pods -o wide
NAME                  READY   STATUS              RESTARTS   AGE     NODE
db-5d8c7b9f6-h7t2m    1/1     Running             0          6d      node-a
db-6f9d7c5b8-n4k2x    0/1     ContainerCreating   0          4m12s   node-b
$ kubectl describe pod db-6f9d7c5b8-n4k2x
Events:
  Type     Reason              Age    From                     Message
  Normal   Scheduled           4m12s  default-scheduler        Successfully assigned default/db-6f9d7c5b8-n4k2x to node-b
  Warning  FailedAttachVolume  4m12s  attachdetach-controller  Multi-Attach error for volume "pvc-3f1c2a9e" Volume is already used by pod(s) db-5d8c7b9f6-h7t2m
  Warning  FailedMount         2m9s   kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data]: timed out waiting for the condition
$ kubectl get pvc
NAME   STATUS   VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data   Bound    pvc-3f1c2a9e   20Gi       RWO            standard       6d

Watch it happen

How PersistentVolume issues unfolds

Step 3 · The failure

The old pod still holds the disk

A rolling update starts the new pod before stopping the old one. The new pod lands on node B and asks for the disk, but it is still attached to node A. Every attempt is refused: Multi-Attach error. The new pod never becomes Ready, so the rollout never removes the old pod. Deadlock.

  • attach retry
  • refused
  • waiting
  • running

Where it breaks: The disk and its rules are the constraint: a ReadWriteOnce disk attaches to one node at a time and lives in one zone. The attach/detach controller and the kubelet only report it, as Multi-Attach, FailedAttachVolume or FailedMount events.

In short

What PersistentVolume issues means

A pod with a volume needs three things to happen before its container starts. The PersistentVolumeClaim must be bound to a PersistentVolume, which may first have to be provisioned by a StorageClass. The disk must be attached to the node the pod was scheduled on. And the kubelet must mount it into the pod. Each stage fails differently. A claim that stays Pending has no matching volume or StorageClass, or is waiting on purpose for a pod to be scheduled. A Multi-Attach error means a ReadWriteOnce disk is still attached to another node, typically because the old pod of a rolling update is still running. A volume node affinity conflict means the disk is in one zone and the pod can only be placed in another. FailedMount is the kubelet giving up on waiting for any of these. The pod's events name the stage that is stuck.

In order

Diagnose in 60 seconds

  1. 1
    kubectl get pvc
    Look for
    STATUS: Bound or Pending. Note the ACCESS MODES and the STORAGECLASS.
    What it tells you
    Pending: the problem is binding or provisioning, before any node is involved. Bound: the volume exists, so look at attach and mount.
  2. 2
    kubectl describe pvc <claim>
    Look for
    The Events. “no persistent volumes available for this claim and no storage class is set”, a ProvisioningFailed message from the driver, or “waiting for first consumer to be created before binding”.
    What it tells you
    The first two are real failures. Waiting for first consumer is normal: the claim binds once a pod that uses it is scheduled.
  3. 3
    kubectl describe pod <pod>
    Look for
    The Events, in order: FailedScheduling with “volume node affinity conflict”, FailedAttachVolume with “Multi-Attach error”, or FailedMount.
    What it tells you
    Names the stage that is stuck: placement (zone), attach (another node holds the disk), or mount (on the node itself).
  4. 4
    kubectl get volumeattachment
    Look for
    Which NODE each volume is attached to, and whether ATTACHED is true.
    What it tells you
    Rules in a disk still held by another node, including a node that is NotReady or gone.
  5. 5
    kubectl get pv <volume> -o jsonpath='{.spec.nodeAffinity}{"\n"}'
    Look for
    The zone the volume is tied to. Compare with kubectl get nodes -L topology.kubernetes.io/zone.
    What it tells you
    Rules in a zone mismatch: the pod can only run where the disk lives.
  6. 6
    kubectl get storageclass
    Look for
    Whether one class is marked (default), its PROVISIONER and its VOLUMEBINDINGMODE.
    What it tells you
    No default class explains claims that never provision. Immediate binding explains disks created in a zone the pod cannot reach.

Most common first

Root causes

  1. 1A ReadWriteOnce volume is still attached to another node: a Deployment's rolling update, or a pod rescheduled while the old one lingers.

    Confirm
    FailedAttachVolume with “Multi-Attach error”, and kubectl get volumeattachment shows the other node.
    Fix now
    Remove the old pod so the disk is released. For a Deployment, switch to strategy Recreate.
    Prevent
    Use a StatefulSet, or Recreate, for anything that owns a single-writer disk.
  2. 2The disk is in one zone and the pod can only be scheduled in another.

    Confirm
    The pod is Pending with “volume node affinity conflict”, and the volume's node affinity names a zone with no usable node.
    Fix now
    Add capacity in the disk's zone, or restore the data to a new volume in a zone that has nodes.
    Prevent
    Use a StorageClass with volumeBindingMode WaitForFirstConsumer, so the disk is created where the pod lands.
  3. 3The claim cannot be provisioned: no default StorageClass, a class name that does not exist, or the CSI driver is not installed.

    Confirm
    The PVC stays Pending, and its events mention no storage class or a provisioning failure.
    Fix now
    Name an existing StorageClass in the claim, mark one as default, or install the driver.
    Prevent
    Check kubectl get storageclass on every new cluster before deploying stateful workloads.
  4. 4The node that held the disk died, and the volume has not been detached from it.

    Confirm
    The VolumeAttachment still points at a NotReady node, and the replacement pod waits in ContainerCreating.
    Fix now
    Confirm the machine is really off, then let the forced detach run or mark the node out-of-service.
    Prevent
    Plan for several minutes of failover for single-writer volumes, or replicate at the application level.
  5. 5The mount itself fails on the node: wrong filesystem, a missing Secret for the driver, or permissions on the volume.

    Confirm
    FailedMount with a specific message from the kubelet or driver, with the volume already attached.
    Fix now
    Fix what the message names: the Secret, the fsType, or the pod's fsGroup.
    Prevent
    Test a new StorageClass with a throwaway pod before pointing real workloads at it.

The tempting wrong fix

Delete the PVC and create it again

Dynamically provisioned volumes usually have reclaim policy Delete, so removing the claim deletes the disk and the data on it. The claim was not the problem: it was Bound. The disk was attached somewhere else or lived in another zone, and a new empty volume hides that while losing everything stored.

Before / after

The most common fix: do not roll a single-writer disk

Before
kind: Deployment
spec:
  replicas: 1
  strategy:
    type: RollingUpdate   # starts the new pod first
  template:
    spec:
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data   # ReadWriteOnce
After
kind: Deployment
spec:
  replicas: 1
  strategy:
    type: Recreate        # stop the old pod, then start the new
  template:
    spec:
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data   # ReadWriteOnce

ReadWriteOnce allows several pods on the same node, so a rolling update can appear to work until the new pod lands on a different node. A StatefulSet is the better home for a pod that owns a disk.

In the real world

Depends on version and provider

Almost everything here depends on the CSI driver and the provider. Which access modes exist differs per driver: block disks such as EBS, Persistent Disk and Azure Disk are single-node, while file storage supports ReadWriteMany. Default StorageClasses on GKE, EKS and AKS typically use WaitForFirstConsumer, but clusters you build yourself may have no default class at all, and on EKS the EBS CSI driver has to be installed as an add-on. How long a disk stays attached to a dead node before a forced detach is a controller setting, several minutes upstream. ReadWriteOncePod restricts a volume to a single pod and needs driver support.

Seen in a real outage

A zonal disk ties a workload to one place, however many nodes you run. At Cloudflare in 2023, services built to span three data centres went down when one lost power, because clusters they depended on existed only in that building.

Break it: Highly available, in one building

Your turn

What would you do?

A single-replica Deployment uses a ReadWriteOnce volume. After a new image is rolled out, the new pod sits in ContainerCreating on another node and the old pod keeps running. What is happening?

PersistentVolume issues questions

Why is my pod stuck in ContainerCreating with a volume?
The container cannot start until its volume is attached to the node and mounted. Run kubectl describe pod and read the events: a Multi-Attach error means another node still holds the disk, and FailedMount means the kubelet timed out waiting for it.
What does Multi-Attach error for volume mean?
The volume is ReadWriteOnce, so it can be attached to one node at a time, and it is still attached to a different node than the one the new pod was scheduled on. Releasing it there, usually by removing the old pod, lets the attach succeed.
Why does my PVC stay Pending?
Either nothing can satisfy it, because there is no matching volume, no StorageClass or a failing provisioner, or it is waiting on purpose. With WaitForFirstConsumer the claim binds only after a pod that uses it has been scheduled.

Sources: Kubernetes docs: Persistent volumes, access modes ↗ · Kubernetes docs: Storage classes, volume binding mode ↗ · Kubernetes docs: Node shutdowns, non-graceful shutdown and forced detach ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play