Skip to content

What's the difference?

OOMKilled vs Evicted

Two ways to lose a pod to memory. Watch which component acts, what it removes, and what comes back.

Scenario 1 · OOMKilled

A container hits its own limit

Pod A's memory climbs to its 512Mi limit. The kernel kills the process with SIGKILL, exit code 137. The pod itself stays on the node, and the kubelet restarts the container in place. The other pods are not involved.

  • memory
  • kill
  • restart
  1. OOMKilledThe kernel lights up. One container dies and the kubelet starts it again in the same pod.
  2. EvictedThe kubelet lights up. A whole pod is removed and a replacement is created somewhere else.

How to tell them apart

Who acts

OOMKilled
The Linux kernel's OOM killer
Evicted
The kubelet on that node

What triggers it

OOMKilled
One container reaches its own memory limit
Evicted
Free memory on the node drops below the kubelet's threshold

What is removed

OOMKilled
The container's process
Evicted
The whole pod, all containers

What comes back

OOMKilled
The same pod; the container is restarted in place
Evicted
A new pod from the controller, possibly on another node

What you see

OOMKilled
Last State: OOMKilled, exit code 137, RESTARTS rising
Evicted
STATUS Evicted, phase Failed, “The node was low on resource: memory”

Who is chosen

OOMKilled
The container that exceeded its limit
Evicted
Pods using most above their requests, lower priority first

First check

OOMKilled
kubectl describe pod: Last State and Limits
Evicted
kubectl describe node: MemoryPressure and events

In short

OOMKilled vs Evicted: the difference

OOMKilled: one container broke its own limit, and the kernel killed it. Evicted: the node ran short, and the kubelet removed a whole pod.

The two look alike from a distance and are different events. OOMKilled is local to one container: its memory use reached the limit set in its own spec, and the Linux kernel killed the process. The pod stays where it is, the kubelet restarts the container in place, and the restart count goes up. Eviction is about the node: free memory on the machine fell below the kubelet's threshold, so the kubelet chose a pod and removed all of it. An evicted pod is not restarted on that node. It is left in the Failed phase with the reason Evicted, and its controller creates a replacement that the scheduler places wherever there is room. A pod can be evicted while well under its own limit, because the kubelet ranks pods by how far their usage exceeds their requests.

Your turn

Which one is this?

Which one is this? A pod with a 2Gi memory limit is using 600Mi when it disappears. kubectl get pods shows it as Failed, and a new pod with a different name is starting on another node.

Questions

What is the difference between OOMKilled and Evicted?
OOMKilled means one container exceeded its own memory limit and the kernel killed it; the kubelet restarts it in the same pod. Evicted means the node ran low on a resource and the kubelet removed the whole pod; a controller creates a replacement.
Can a pod be evicted while under its memory limit?
Yes. Eviction is decided at node level. The kubelet looks at how much each pod uses above its requests, and at pod priority, not at whether a limit was reached.
How do I avoid evictions?
Set memory requests close to real usage so the scheduler does not overfill nodes, and keep limits near requests for important workloads. Pods whose usage stays within their requests are the last to be evicted.

Sources: Kubernetes docs: Node-pressure eviction ↗ · Kubernetes docs: Resource management for pods and containers ↗ · Report a problem

Diagrams are our own simulations. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play