Skip to content
Breaks at SchedulerDegrades the serviceVery common

Pending pod

The pod has been accepted but no node can take it, so it has not been scheduled and nothing is running.

What you'll seeIllustrative output
$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP       NODE
report-5b7c9d6f8-hl2mq   0/1     Pending   0          4m37s   <none>   <none>
$ kubectl describe pod report-5b7c9d6f8-hl2mq
Status:       Pending
Node:         <none>
Node-Selectors:  disktype=ssd
    Requests:
      cpu:     2
      memory:  1Gi
Events:
  Type     Reason            Age    From               Message
  Warning  FailedScheduling  4m37s  default-scheduler  0/3 nodes are available: 1 Insufficient cpu, 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint {dedicated: gpu}. preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling.

Watch it happen

How Pending pod unfolds

Step 3 · The failure

Every node says no

Now the pod requests 2 CPUs and demands an ssd node. A has the disk but not the CPU. B has the CPU but not the disk. C is tainted. 0 of 3 nodes fit, so the scheduler lights up and the pod stays Pending. It will ask again later and get the same answers.

  • does it fit?
  • rejected
  • retry

Where it breaks: The scheduler is where it stops: it checks every node against what the pod asks for and finds none that fits. No kubelet is involved yet, because the pod has not been given to any node.

In short

What Pending pod means

A pod stays Pending when the scheduler cannot find a node for it. For every new pod the scheduler filters the nodes: does this node have enough unreserved CPU and memory for the pod's requests, do its labels match the pod's node selector or affinity, does the pod tolerate the node's taints, can the pod's volumes be used there. A node that fails any one check is out. If at least one node is left, the pod is bound to the best of them and the kubelet on that node takes over. If none is left, the pod waits and the scheduler tries again when something in the cluster changes. The reason is never hidden: the FailedScheduling event counts how many nodes were rejected and why. Note that the scheduler compares requests, not live usage, so a node can look idle and still be full.

In order

Diagnose in 60 seconds

  1. 1
    kubectl get pod <pod> -o wide
    Look for
    STATUS Pending with NODE <none>.
    What it tells you
    Rules in: the pod was never scheduled. If a node is listed and the status is ContainerCreating, scheduling worked and the problem is on that node.
  2. 2
    kubectl describe pod <pod>
    Look for
    The FailedScheduling event. It counts the nodes rejected for each reason.
    What it tells you
    “Insufficient cpu” or “Insufficient memory”: requests do not fit. “didn't match Pod's node affinity/selector”: labels. “untolerated taint”: taints. “unbound immediate PersistentVolumeClaims”: storage. No event at all: the scheduler itself may not be running.
  3. 3
    kubectl describe nodes | grep -A 8 'Allocated resources'
    Look for
    Per node, the share of CPU and memory already requested by pods.
    What it tells you
    Rules in a full cluster. A node near 100% requested has no room for new requests, whatever its real load is.
  4. 4
    kubectl get nodes --show-labels
    Look for
    Whether any node carries the labels named in the pod's nodeSelector or affinity. Check for SchedulingDisabled too.
    What it tells you
    Rules in a selector that matches nothing, or nodes that were cordoned.
  5. 5
    kubectl describe nodes | grep -i taints
    Look for
    The taints on each node, compared with the pod's tolerations.
    What it tells you
    Rules in a taint the pod does not tolerate, such as a dedicated pool or a node marked not-ready.
  6. 6
    kubectl get pvc
    Look for
    Any claim used by the pod that is itself Pending.
    What it tells you
    Rules in storage: a pod cannot be placed until its claims can be bound. See the PersistentVolume page.

Most common first

Root causes

  1. 1Not enough unreserved CPU or memory on any node for the pod's requests.

    Confirm
    The event says “Insufficient cpu” or “Insufficient memory”, and Allocated resources on the nodes is near 100%.
    Fix now
    Add nodes, free capacity by scaling something down, or lower requests that are larger than real usage.
    Prevent
    Set requests from measured usage and run a cluster autoscaler so capacity follows demand.
  2. 2A nodeSelector or node affinity that no node satisfies.

    Confirm
    The event says nodes “didn't match Pod's node affinity/selector”, and no node has the label.
    Fix now
    Correct the selector, or label a suitable node.
    Prevent
    Prefer soft (preferred) affinity unless the constraint is truly mandatory.
  3. 3Every candidate node has a taint the pod does not tolerate.

    Confirm
    The event names an “untolerated taint” and its key.
    Fix now
    Add the matching toleration to the pod, or remove the taint if it is no longer meant to be there.
    Prevent
    Document which node pools are dedicated, and give their workloads the toleration in a shared template.
  4. 4A PersistentVolumeClaim the pod uses cannot be bound, or its volume lives in a zone with no suitable node.

    Confirm
    The event mentions unbound PersistentVolumeClaims or a volume node affinity conflict, and kubectl get pvc shows Pending.
    Fix now
    Fix the claim: its StorageClass, size or zone.
    Prevent
    Use a StorageClass with WaitForFirstConsumer so the volume is created where the pod is scheduled.
  5. 5Pod anti-affinity or topology spread rules leave no legal node.

    Confirm
    The event says nodes “didn't match pod anti-affinity rules” or “didn't match pod topology spread constraints”.
    Fix now
    Relax the rule, or add nodes in the missing zone.
    Prevent
    Check that replica counts and spread rules can be satisfied by the number of nodes and zones you have.

The tempting wrong fix

Delete the pod and let it be recreated

A new pod carries the same requests, selectors and tolerations and meets the same nodes, so the scheduler reaches the same verdict. The scheduler was already retrying by itself. Nothing changes until either the pod's demands or the cluster's capacity does.

Before / after

The most common fix: requests that match what the pod uses

Before
resources:
  requests:
    cpu: "2"        # copied from another service
    memory: 1Gi
# event: 0/3 nodes are available: Insufficient cpu
After
resources:
  requests:
    cpu: 500m       # measured: about 0.4 CPU at peak
    memory: 1Gi

Only lower a request to what the pod really uses. A request that is too small gets the pod scheduled onto a node where it will then be starved or evicted.

In the real world

Depends on version and provider

What happens next depends on the cluster. With a cluster autoscaler, or Karpenter, a pod that is Pending for lack of resources triggers a new node, so it may sit Pending for a minute or two and then start; look for TriggeredScaleUp or NotTriggerScaleUp events. Without one it waits for ever. Managed services reserve part of each node for the system, so allocatable CPU and memory are lower than the machine size. GKE Autopilot and Fargate profiles create capacity per pod and report different events.

Your turn

What would you do?

A pod is Pending. kubectl top nodes shows every node under 20% CPU. The event reads “0/5 nodes are available: 5 Insufficient cpu”. How can both be true?

Pending pod questions

Why is my Kubernetes pod stuck in Pending?
Because the scheduler cannot find a node that satisfies it. The usual reasons are CPU or memory requests that do not fit on any node, a node selector or affinity that matches no node, taints the pod does not tolerate, or a volume claim that cannot be bound.
How do I see why a pod is Pending?
Run kubectl describe pod and read the FailedScheduling event at the bottom. It states how many nodes were considered and how many were rejected for each reason.
Does the scheduler use real CPU usage?
No. It uses the resource requests declared by pods. A node is full when the requests of its pods add up to its allocatable capacity, even if those pods are idle.

Sources: Kubernetes docs: Kubernetes scheduler ↗ · Kubernetes docs: Debug pods, my pod stays pending ↗ · Kubernetes docs: Taints and tolerations ↗ · Kubernetes docs: Assigning pods to nodes ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play