Pending pod
The pod has been accepted but no node can take it, so it has not been scheduled and nothing is running.
$ kubectl get pods -o wideNAME READY STATUS RESTARTS AGE IP NODE report-5b7c9d6f8-hl2mq 0/1 Pending 0 4m37s <none> <none>
$ kubectl describe pod report-5b7c9d6f8-hl2mqStatus: Pending
Node: <none>
Node-Selectors: disktype=ssd
Requests:
cpu: 2
memory: 1Gi
Events:
Type Reason Age From Message
Warning FailedScheduling 4m37s default-scheduler 0/3 nodes are available: 1 Insufficient cpu, 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint {dedicated: gpu}. preemption: 0/3 nodes are available: 1 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling.Watch it happen
How Pending pod unfolds
Step 3 · The failure
Every node says no
Now the pod requests 2 CPUs and demands an ssd node. A has the disk but not the CPU. B has the CPU but not the disk. C is tainted. 0 of 3 nodes fit, so the scheduler lights up and the pod stays Pending. It will ask again later and get the same answers.
- does it fit?
- rejected
- retry
Step 3 · The failure
Every node says no
Now the pod requests 2 CPUs and demands an ssd node. A has the disk but not the CPU. B has the CPU but not the disk. C is tainted. 0 of 3 nodes fit, so the scheduler lights up and the pod stays Pending. It will ask again later and get the same answers.
- does it fit?
- rejected
- retry
Where it breaks: The scheduler is where it stops: it checks every node against what the pod asks for and finds none that fits. No kubelet is involved yet, because the pod has not been given to any node.
In short
What Pending pod means
A pod stays Pending when the scheduler cannot find a node for it. For every new pod the scheduler filters the nodes: does this node have enough unreserved CPU and memory for the pod's requests, do its labels match the pod's node selector or affinity, does the pod tolerate the node's taints, can the pod's volumes be used there. A node that fails any one check is out. If at least one node is left, the pod is bound to the best of them and the kubelet on that node takes over. If none is left, the pod waits and the scheduler tries again when something in the cluster changes. The reason is never hidden: the FailedScheduling event counts how many nodes were rejected and why. Note that the scheduler compares requests, not live usage, so a node can look idle and still be full.
In order
Diagnose in 60 seconds
- 1
kubectl get pod <pod> -o wide- Look for
- STATUS Pending with NODE <none>.
- What it tells you
- Rules in: the pod was never scheduled. If a node is listed and the status is ContainerCreating, scheduling worked and the problem is on that node.
- 2
kubectl describe pod <pod>- Look for
- The FailedScheduling event. It counts the nodes rejected for each reason.
- What it tells you
- “Insufficient cpu” or “Insufficient memory”: requests do not fit. “didn't match Pod's node affinity/selector”: labels. “untolerated taint”: taints. “unbound immediate PersistentVolumeClaims”: storage. No event at all: the scheduler itself may not be running.
- 3
kubectl describe nodes | grep -A 8 'Allocated resources'- Look for
- Per node, the share of CPU and memory already requested by pods.
- What it tells you
- Rules in a full cluster. A node near 100% requested has no room for new requests, whatever its real load is.
- 4
kubectl get nodes --show-labels- Look for
- Whether any node carries the labels named in the pod's nodeSelector or affinity. Check for SchedulingDisabled too.
- What it tells you
- Rules in a selector that matches nothing, or nodes that were cordoned.
- 5
kubectl describe nodes | grep -i taints- Look for
- The taints on each node, compared with the pod's tolerations.
- What it tells you
- Rules in a taint the pod does not tolerate, such as a dedicated pool or a node marked not-ready.
- 6
kubectl get pvc- Look for
- Any claim used by the pod that is itself Pending.
- What it tells you
- Rules in storage: a pod cannot be placed until its claims can be bound. See the PersistentVolume page.
Most common first
Root causes
1Not enough unreserved CPU or memory on any node for the pod's requests.
- Confirm
- The event says “Insufficient cpu” or “Insufficient memory”, and Allocated resources on the nodes is near 100%.
- Fix now
- Add nodes, free capacity by scaling something down, or lower requests that are larger than real usage.
- Prevent
- Set requests from measured usage and run a cluster autoscaler so capacity follows demand.
2A nodeSelector or node affinity that no node satisfies.
- Confirm
- The event says nodes “didn't match Pod's node affinity/selector”, and no node has the label.
- Fix now
- Correct the selector, or label a suitable node.
- Prevent
- Prefer soft (preferred) affinity unless the constraint is truly mandatory.
3Every candidate node has a taint the pod does not tolerate.
- Confirm
- The event names an “untolerated taint” and its key.
- Fix now
- Add the matching toleration to the pod, or remove the taint if it is no longer meant to be there.
- Prevent
- Document which node pools are dedicated, and give their workloads the toleration in a shared template.
4A PersistentVolumeClaim the pod uses cannot be bound, or its volume lives in a zone with no suitable node.
- Confirm
- The event mentions unbound PersistentVolumeClaims or a volume node affinity conflict, and kubectl get pvc shows Pending.
- Fix now
- Fix the claim: its StorageClass, size or zone.
- Prevent
- Use a StorageClass with WaitForFirstConsumer so the volume is created where the pod is scheduled.
5Pod anti-affinity or topology spread rules leave no legal node.
- Confirm
- The event says nodes “didn't match pod anti-affinity rules” or “didn't match pod topology spread constraints”.
- Fix now
- Relax the rule, or add nodes in the missing zone.
- Prevent
- Check that replica counts and spread rules can be satisfied by the number of nodes and zones you have.
The tempting wrong fix
Delete the pod and let it be recreated
Before / after
The most common fix: requests that match what the pod uses
resources:
requests:
cpu: "2" # copied from another service
memory: 1Gi
# event: 0/3 nodes are available: Insufficient cpuresources:
requests:
cpu: 500m # measured: about 0.4 CPU at peak
memory: 1GiOnly lower a request to what the pod really uses. A request that is too small gets the pod scheduled onto a node where it will then be starved or evicted.
In the real world
Depends on version and provider
Your turn
What would you do?
A pod is Pending. kubectl top nodes shows every node under 20% CPU. The event reads “0/5 nodes are available: 5 Insufficient cpu”. How can both be true?
Related errors
Pending pod questions
- Why is my Kubernetes pod stuck in Pending?
- Because the scheduler cannot find a node that satisfies it. The usual reasons are CPU or memory requests that do not fit on any node, a node selector or affinity that matches no node, taints the pod does not tolerate, or a volume claim that cannot be bound.
- How do I see why a pod is Pending?
- Run kubectl describe pod and read the FailedScheduling event at the bottom. It states how many nodes were considered and how many were rejected for each reason.
- Does the scheduler use real CPU usage?
- No. It uses the resource requests declared by pods. A node is full when the requests of its pods add up to its allocatable capacity, even if those pods are idle.
Sources: Kubernetes docs: Kubernetes scheduler ↗ · Kubernetes docs: Debug pods, my pod stays pending ↗ · Kubernetes docs: Taints and tolerations ↗ · Kubernetes docs: Assigning pods to nodes ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.