Skip to content
Breaks at kubelet · Controller managerCan take the service downCommon

Node NotReady

The control plane has stopped hearing from a node's kubelet, so it no longer trusts the node or the pods on it.

What you'll seeIllustrative output
$ kubectl get nodes
NAME     STATUS     ROLES    AGE   VERSION
node-a   Ready      <none>   41d   v1.33.2
node-b   NotReady   <none>   41d   v1.33.2
node-c   Ready      <none>   41d   v1.33.2
$ kubectl describe node node-b
Taints:             node.kubernetes.io/unreachable:NoExecute
                    node.kubernetes.io/unreachable:NoSchedule
Conditions:
  Type             Status    Reason              Message
  MemoryPressure   Unknown   NodeStatusUnknown   Kubelet stopped posting node status.
  DiskPressure     Unknown   NodeStatusUnknown   Kubelet stopped posting node status.
  Ready            Unknown   NodeStatusUnknown   Kubelet stopped posting node status.
Events:
  Type     Reason         Age   From             Message
  Normal   NodeNotReady   3m    node-controller  Node node-b status is now: NodeNotReady
$ kubectl get pods -o wide
NAME                  READY   STATUS        RESTARTS   AGE     NODE
api-7d9c5b6f4-x2k8p   1/1     Terminating   0          2d4h    node-b
api-7d9c5b6f4-r8m3j   0/1     Pending       0          4s      <none>

Watch it happen

How Node NotReady unfolds

Step 3 · The failure

The pod waits five more minutes

The node is NotReady, but its pod is not moved yet. Every pod tolerates the taint for 300 seconds by default. Only when that countdown ends is the pod evicted and a replacement scheduled on another node. With one replica, that is about six minutes of outage.

  • silent kubelet / evict
  • tolerating the taint
  • lease / replacement

Where it breaks: The kubelet on the node is what goes quiet: it stops renewing its heartbeat. The node-lifecycle controller in the control plane notices, marks the node NotReady, taints it and, minutes later, evicts its pods.

In short

What Node NotReady means

A node is Ready only as long as its kubelet keeps saying so. Every 10 seconds the kubelet renews a small Lease object in the API server, and it also posts its status. When those updates stop, nothing happens at first. The node-lifecycle controller waits out a grace period, 50 seconds by default, then sets the node's Ready condition to Unknown, which kubectl shows as NotReady, and adds taints that keep new pods away. Pods already on the node are not moved yet either: every pod tolerates those taints for 300 seconds by default. Only after that are they evicted, and their controllers create replacements on other nodes. So a single-replica service on a dead node is missing for about six minutes. The node itself may be fine; a kubelet that has crashed, lost its network path or run out of resources looks exactly the same from the control plane.

In order

Diagnose in 60 seconds

  1. 1
    kubectl get nodes
    Look for
    Which node is NotReady, and whether it is one node or many at once.
    What it tells you
    One node points at that machine or its kubelet. Many at once points at something shared: the network, the API server, a rollout to every node.
  2. 2
    kubectl describe node <node>
    Look for
    The Ready condition. Status Unknown with “Kubelet stopped posting node status”, or Status False with a reason. Then the pressure conditions and the Taints line.
    What it tells you
    Unknown: the control plane cannot hear the kubelet at all. False: the kubelet is alive and reports a problem itself, such as the runtime or the CNI plugin not being ready.
  3. 3
    kubectl get lease -n kube-node-lease <node> -o jsonpath='{.spec.renewTime}{"\n"}'
    Look for
    The time of the last heartbeat, compared with now.
    What it tells you
    Tells you exactly when the kubelet went quiet, so you can line it up with a deploy, a reboot or a network change.
  4. 4
    kubectl get pods -A -o wide --field-selector spec.nodeName=<node>
    Look for
    What was running there, and which pods are Terminating or already replaced.
    What it tells you
    Shows the blast radius. StatefulSet pods stuck Terminating are not replaced until the old pod is confirmed gone.
  5. 5
    journalctl -u kubelet --since "15 min ago"
    Look for
    Run on the node itself, over SSH or the provider's console. Look for certificate errors, the API server being unreachable, a runtime that is not responding, or out-of-memory kills.
    What it tells you
    Rules in the actual cause. If you cannot reach the node at all, the machine or its network is down.

Most common first

Root causes

  1. 1The kubelet or the container runtime has stopped: crashed, hung, or starved because the node ran out of memory, disk or process IDs.

    Confirm
    The machine answers over SSH, but systemctl status kubelet or the runtime shows it failed, or the kernel log shows out-of-memory kills.
    Fix now
    Restart the kubelet or the runtime, or reboot the node. Free the exhausted resource first.
    Prevent
    Reserve resources for the system (kubeReserved, systemReserved) and give every pod requests and limits.
  2. 2The node lost its network path to the API server: a route, firewall rule, security group or CNI change.

    Confirm
    The kubelet is running, and its log is full of timeouts talking to the API server. Often several nodes fail together.
    Fix now
    Restore the network path. The node becomes Ready on its own once heartbeats arrive again.
    Prevent
    Roll out network and operating system changes to a few nodes at a time.
  3. 3The machine itself is gone: hardware failure, a reclaimed spot instance, a host reboot.

    Confirm
    The cloud console shows the instance stopped or terminated, and the node cannot be reached at all.
    Fix now
    Replace the machine. A node group or auto-repair feature normally does this for you.
    Prevent
    Run at least two replicas spread across nodes and zones, with a PodDisruptionBudget.
  4. 4The kubelet is alive but reports NotReady: the container runtime is down or the CNI plugin is not initialised.

    Confirm
    The Ready condition is False, not Unknown, with a reason such as KubeletNotReady and a message naming the runtime or network plugin.
    Fix now
    Fix the named component: restart the runtime, or repair or reinstall the CNI plugin's pods on that node.
    Prevent
    Treat the CNI plugin and runtime as critical add-ons, and upgrade them node by node.
  5. 5The kubelet's client certificate has expired or can no longer be rotated.

    Confirm
    The kubelet log shows authentication or x509 errors against the API server.
    Fix now
    Renew or re-issue the kubelet's certificate and approve any pending certificate requests.
    Prevent
    Keep certificate rotation enabled and alert on certificates close to expiry.

The tempting wrong fix

Force-delete the stuck pods

kubectl delete pod --force only removes the record. If the node is cut off but still running, the old container keeps going, and for a StatefulSet a second copy now starts with the same identity and may write to the same volume. First make sure the machine is really off, or fence it, and only then clear the pods.

Before / after

The most common fix: two spread replicas and a shorter toleration

Before
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: api
          image: registry.example.com/api:1.4.2
      # no tolerations: Kubernetes adds 300s
      # for not-ready and unreachable
After
spec:
  replicas: 2
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: api }
      tolerations:
        - key: node.kubernetes.io/unreachable
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 30
        - key: node.kubernetes.io/not-ready
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 30
      containers:
        - name: api
          image: registry.example.com/api:1.4.2

A short toleration suits stateless pods. Keep the default, or longer, for pods that are slow or risky to start a second copy of. The replicas matter more than the number: with two, the wait is not an outage.

In the real world

Depends on version and provider

The 50 second grace period (node-monitor-grace-period) and the 300 second default toleration are upstream defaults; older releases used 40 seconds, and on managed control planes you usually cannot change the grace period. Providers add their own layer: GKE and AKS node auto-repair and EKS managed node groups replace unhealthy machines, and when the cloud reports a machine as deleted the Node object is removed and its pods are cleaned up sooner. DaemonSet pods tolerate these taints indefinitely and are never evicted for them. For StatefulSets on a node that is known to be shut down, the out-of-service taint lets volumes detach and pods move.

Seen in a real outage

Nodes rarely fail alone when they share a schedule. At Datadog in 2023 an automatic system update restarted the network service on tens of thousands of nodes in the same window and removed the routes their container network needed, so they dropped off the network together.

Break it: 06:00 UTC, everywhere

Your turn

What would you do?

A node loses power at 12:00:00. Its only replica of a Deployment has no custom tolerations. With upstream defaults, roughly when is the replacement pod created?

Node NotReady questions

What does NotReady mean for a Kubernetes node?
The control plane is not getting healthy status from that node's kubelet. Either the kubelet has stopped reporting, shown as Ready Unknown, or it reports a problem itself, shown as Ready False. New pods are kept away from the node.
How long before pods are moved off a NotReady node?
With upstream defaults, about six minutes. The node is marked NotReady after roughly 50 seconds without heartbeats, and pods then tolerate the resulting taint for 300 seconds before they are evicted. Both values can be changed.
Why are my pods stuck Terminating on a NotReady node?
Deleting a pod needs the kubelet to confirm the containers have stopped, and that kubelet is unreachable. Deployments start replacements anyway. StatefulSets wait, because starting a second copy could be unsafe.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play