Node NotReady
The control plane has stopped hearing from a node's kubelet, so it no longer trusts the node or the pods on it.
$ kubectl get nodesNAME STATUS ROLES AGE VERSION node-a Ready <none> 41d v1.33.2 node-b NotReady <none> 41d v1.33.2 node-c Ready <none> 41d v1.33.2
$ kubectl describe node node-bTaints: node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unreachable:NoSchedule
Conditions:
Type Status Reason Message
MemoryPressure Unknown NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown NodeStatusUnknown Kubelet stopped posting node status.
Events:
Type Reason Age From Message
Normal NodeNotReady 3m node-controller Node node-b status is now: NodeNotReady$ kubectl get pods -o wideNAME READY STATUS RESTARTS AGE NODE api-7d9c5b6f4-x2k8p 1/1 Terminating 0 2d4h node-b api-7d9c5b6f4-r8m3j 0/1 Pending 0 4s <none>
Watch it happen
How Node NotReady unfolds
Step 3 · The failure
The pod waits five more minutes
The node is NotReady, but its pod is not moved yet. Every pod tolerates the taint for 300 seconds by default. Only when that countdown ends is the pod evicted and a replacement scheduled on another node. With one replica, that is about six minutes of outage.
- silent kubelet / evict
- tolerating the taint
- lease / replacement
Step 3 · The failure
The pod waits five more minutes
The node is NotReady, but its pod is not moved yet. Every pod tolerates the taint for 300 seconds by default. Only when that countdown ends is the pod evicted and a replacement scheduled on another node. With one replica, that is about six minutes of outage.
- silent kubelet / evict
- tolerating the taint
- lease / replacement
Where it breaks: The kubelet on the node is what goes quiet: it stops renewing its heartbeat. The node-lifecycle controller in the control plane notices, marks the node NotReady, taints it and, minutes later, evicts its pods.
In short
What Node NotReady means
A node is Ready only as long as its kubelet keeps saying so. Every 10 seconds the kubelet renews a small Lease object in the API server, and it also posts its status. When those updates stop, nothing happens at first. The node-lifecycle controller waits out a grace period, 50 seconds by default, then sets the node's Ready condition to Unknown, which kubectl shows as NotReady, and adds taints that keep new pods away. Pods already on the node are not moved yet either: every pod tolerates those taints for 300 seconds by default. Only after that are they evicted, and their controllers create replacements on other nodes. So a single-replica service on a dead node is missing for about six minutes. The node itself may be fine; a kubelet that has crashed, lost its network path or run out of resources looks exactly the same from the control plane.
In order
Diagnose in 60 seconds
- 1
kubectl get nodes- Look for
- Which node is NotReady, and whether it is one node or many at once.
- What it tells you
- One node points at that machine or its kubelet. Many at once points at something shared: the network, the API server, a rollout to every node.
- 2
kubectl describe node <node>- Look for
- The Ready condition. Status Unknown with “Kubelet stopped posting node status”, or Status False with a reason. Then the pressure conditions and the Taints line.
- What it tells you
- Unknown: the control plane cannot hear the kubelet at all. False: the kubelet is alive and reports a problem itself, such as the runtime or the CNI plugin not being ready.
- 3
kubectl get lease -n kube-node-lease <node> -o jsonpath='{.spec.renewTime}{"\n"}'- Look for
- The time of the last heartbeat, compared with now.
- What it tells you
- Tells you exactly when the kubelet went quiet, so you can line it up with a deploy, a reboot or a network change.
- 4
kubectl get pods -A -o wide --field-selector spec.nodeName=<node>- Look for
- What was running there, and which pods are Terminating or already replaced.
- What it tells you
- Shows the blast radius. StatefulSet pods stuck Terminating are not replaced until the old pod is confirmed gone.
- 5
journalctl -u kubelet --since "15 min ago"- Look for
- Run on the node itself, over SSH or the provider's console. Look for certificate errors, the API server being unreachable, a runtime that is not responding, or out-of-memory kills.
- What it tells you
- Rules in the actual cause. If you cannot reach the node at all, the machine or its network is down.
Most common first
Root causes
1The kubelet or the container runtime has stopped: crashed, hung, or starved because the node ran out of memory, disk or process IDs.
- Confirm
- The machine answers over SSH, but systemctl status kubelet or the runtime shows it failed, or the kernel log shows out-of-memory kills.
- Fix now
- Restart the kubelet or the runtime, or reboot the node. Free the exhausted resource first.
- Prevent
- Reserve resources for the system (kubeReserved, systemReserved) and give every pod requests and limits.
2The node lost its network path to the API server: a route, firewall rule, security group or CNI change.
- Confirm
- The kubelet is running, and its log is full of timeouts talking to the API server. Often several nodes fail together.
- Fix now
- Restore the network path. The node becomes Ready on its own once heartbeats arrive again.
- Prevent
- Roll out network and operating system changes to a few nodes at a time.
3The machine itself is gone: hardware failure, a reclaimed spot instance, a host reboot.
- Confirm
- The cloud console shows the instance stopped or terminated, and the node cannot be reached at all.
- Fix now
- Replace the machine. A node group or auto-repair feature normally does this for you.
- Prevent
- Run at least two replicas spread across nodes and zones, with a PodDisruptionBudget.
4The kubelet is alive but reports NotReady: the container runtime is down or the CNI plugin is not initialised.
- Confirm
- The Ready condition is False, not Unknown, with a reason such as KubeletNotReady and a message naming the runtime or network plugin.
- Fix now
- Fix the named component: restart the runtime, or repair or reinstall the CNI plugin's pods on that node.
- Prevent
- Treat the CNI plugin and runtime as critical add-ons, and upgrade them node by node.
5The kubelet's client certificate has expired or can no longer be rotated.
- Confirm
- The kubelet log shows authentication or x509 errors against the API server.
- Fix now
- Renew or re-issue the kubelet's certificate and approve any pending certificate requests.
- Prevent
- Keep certificate rotation enabled and alert on certificates close to expiry.
The tempting wrong fix
Force-delete the stuck pods
Before / after
The most common fix: two spread replicas and a shorter toleration
spec:
replicas: 1
template:
spec:
containers:
- name: api
image: registry.example.com/api:1.4.2
# no tolerations: Kubernetes adds 300s
# for not-ready and unreachablespec:
replicas: 2
template:
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels: { app: api }
tolerations:
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 30
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 30
containers:
- name: api
image: registry.example.com/api:1.4.2A short toleration suits stateless pods. Keep the default, or longer, for pods that are slow or risky to start a second copy of. The replicas matter more than the number: with two, the wait is not an outage.
In the real world
Depends on version and provider
Seen in a real outage
Nodes rarely fail alone when they share a schedule. At Datadog in 2023 an automatic system update restarted the network service on tens of thousands of nodes in the same window and removed the routes their container network needed, so they dropped off the network together.
Break it: 06:00 UTC, everywhereYour turn
What would you do?
A node loses power at 12:00:00. Its only replica of a Deployment has no custom tolerations. With upstream defaults, roughly when is the replacement pod created?
Related errors
Node NotReady questions
- What does NotReady mean for a Kubernetes node?
- The control plane is not getting healthy status from that node's kubelet. Either the kubelet has stopped reporting, shown as Ready Unknown, or it reports a problem itself, shown as Ready False. New pods are kept away from the node.
- How long before pods are moved off a NotReady node?
- With upstream defaults, about six minutes. The node is marked NotReady after roughly 50 seconds without heartbeats, and pods then tolerate the resulting taint for 300 seconds before they are evicted. Both values can be changed.
- Why are my pods stuck Terminating on a NotReady node?
- Deleting a pod needs the kubelet to confirm the containers have stopped, and that kubelet is unreachable. Deployments start replacements anyway. StatefulSets wait, because starting a second copy could be unsafe.
Sources: Kubernetes docs: Nodes, heartbeats and the node controller ↗ · Kubernetes docs: Taints and tolerations, taint-based evictions ↗ · Kubernetes docs: Leases ↗ · Kubernetes docs: kube-controller-manager reference (node-monitor-grace-period) ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.