OOMKilled
The container used more memory than its limit, and the Linux kernel killed it. Exit code 137.
$ kubectl get podsNAME READY STATUS RESTARTS AGE worker-5c8f7d9b6-qm4zt 0/1 OOMKilled 3 (41s ago) 6m12s
$ kubectl describe pod worker-5c8f7d9b6-qm4zt State: Terminated
Reason: OOMKilled
Exit Code: 137
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Restart Count: 3
Limits:
memory: 512Mi
Requests:
memory: 256Mi$ kubectl top pod worker-5c8f7d9b6-qm4zt --containersPOD NAME CPU(cores) MEMORY(bytes) worker-5c8f7d9b6-qm4zt worker 212m 498Mi
Watch it happen
How OOMKilled unfolds
Step 3 · The failure
Restart, climb, killed again
The restarted container does the same work and climbs to the limit again. Each kill is followed by a longer back-off, so the pod drifts into CrashLoopBackOff. The kernel lights up each time: it is doing exactly what the limit told it to.
- memory in use
- kill
- restart
Step 3 · The failure
Restart, climb, killed again
The restarted container does the same work and climbs to the limit again. Each kill is followed by a longer back-off, so the pod drifts into CrashLoopBackOff. The kernel lights up each time: it is doing exactly what the limit told it to.
- memory in use
- kill
- restart
Where it breaks: The Linux kernel does the killing, enforcing the memory limit you set on the container. Kubernetes only records the reason and restarts the container.
In short
What OOMKilled means
OOMKilled means the container tried to use more memory than its limit allows. The limit is not enforced by Kubernetes itself but by the Linux kernel, through the container's cgroup. When usage reaches the limit and nothing can be freed, the kernel's out-of-memory killer ends the process with SIGKILL. The process gets no warning and writes no error, which is why the application logs simply stop. Kubernetes records Reason OOMKilled with exit code 137 and the kubelet restarts the container, so repeated kills show up as CrashLoopBackOff. This is different from eviction, where the node as a whole is short of memory and the kubelet removes entire pods. The question that decides the fix is whether memory levels off below some value or climbs without end.
In order
Diagnose in 60 seconds
- 1
kubectl describe pod <pod>- Look for
- Last State: Reason OOMKilled, Exit Code 137. Note the memory limit listed under Limits.
- What it tells you
- Rules in: this container hit its own limit. If the reason is Error or Completed, it is not an OOM kill.
- 2
kubectl get pod <pod> -o jsonpath='{.status.reason}{"\n"}'- Look for
- An empty result, or the word Evicted.
- What it tells you
- Evicted means the node ran short of memory and the kubelet removed the whole pod. That is a node problem, not this container's limit.
- 3
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}'- Look for
- The single word OOMKilled.
- What it tells you
- The same fact as describe, in a form you can script or run across many pods.
- 4
kubectl top pod <pod> --containers- Look for
- Current memory per container against the limit. Run it a few times as the container warms up. Needs metrics-server.
- What it tells you
- Rules in which container is growing, and how close to the limit it sits before each kill.
- 5
kubectl logs <pod> --previous- Look for
- What the app was doing just before the logs stop. There will be no error line: SIGKILL cannot be caught.
- What it tells you
- Points to the request, job or batch that was running when memory ran out.
- 6
kubectl describe node <node>- Look for
- The MemoryPressure condition, and SystemOOM or eviction events.
- What it tells you
- Rules in a node-level shortage affecting several pods, instead of one container's limit.
Most common first
Root causes
1The limit is lower than what the app really needs under load.
- Confirm
- Memory rises to roughly the same level every time and the kill comes at the limit, often during traffic peaks.
- Fix now
- Raise the limit above the measured peak, and raise the request with it.
- Prevent
- Size limits from load tests or from usage history, not from a guess.
2A memory leak: usage grows for as long as the container runs.
- Confirm
- The time between kills is long and regular, and usage never levels off whatever the load.
- Fix now
- Find and fix the leak. A higher limit only spaces the kills further apart.
- Prevent
- Alert on memory that keeps rising over hours, before it reaches the limit.
3A JVM (or another runtime) whose heap is sized as if it owned the whole container. A Java process also needs memory outside the heap: metaspace, thread stacks, direct buffers and other native memory.
- Confirm
- The maximum heap (-Xmx, or the share set by -XX:MaxRAMPercentage) plus that non-heap memory adds up to more than the container limit.
- Fix now
- Cap the heap at a share of the limit and leave the rest free, for example JAVA_TOOL_OPTIONS="-XX:MaxRAMPercentage=70". Never set -Xmx equal to the limit.
- Prevent
- Derive heap settings from the container's limit, and watch non-heap memory as well as heap in your JVM metrics.
4One large piece of work: a big upload, an export or a batch loaded into memory at once.
- Confirm
- Kills line up with particular requests or jobs, and the previous logs end in the middle of one.
- Fix now
- Stream or page the data, or move that work to a job with its own, larger limit.
- Prevent
- Put a size cap on inputs that are held in memory.
5Files written to a memory-backed volume count against the container's memory.
- Confirm
- The pod uses an emptyDir with medium: Memory, and the kills follow large writes to it.
- Fix now
- Set a sizeLimit on the volume, or use a disk-backed emptyDir.
- Prevent
- Treat memory-backed volumes as part of the memory budget.
The tempting wrong fix
Raise the memory limit
Before / after
The most common fix: a limit that fits the measured peak
resources:
requests:
memory: 256Mi
limits:
memory: 512Mi # the app peaks at about 520Miresources:
requests:
memory: 768Mi # equal to the limit: the scheduler reserves it
limits:
memory: 768Mi # measured peak plus headroomSetting the memory request equal to the limit means the node always has that memory set aside for the pod, so it is among the last candidates when the node runs short.
In the real world
Depends on version and provider
Seen in a real outage
More memory buys time, not a fix, when something keeps allocating. At PagerDuty in 2025, Kafka brokers ran out of heap one after another; raising the heap only delayed the next failure.
Break it: A producer for every requestYour turn
What would you do?
A container is OOMKilled about every six hours, whatever the traffic. Its memory graph is a steady upward line that never flattens. What does that tell you?
Related errors
OOMKilled questions
- What does OOMKilled mean in Kubernetes?
- The container used more memory than its memory limit. The Linux kernel killed the process to enforce the limit, and Kubernetes recorded the reason as OOMKilled.
- What is exit code 137?
- It means the process was ended by SIGKILL: 128 plus signal number 9. With Reason OOMKilled it was the kernel's out-of-memory killer. Without it, something else sent the kill, such as the kubelet after a failed liveness probe.
- What is the difference between OOMKilled and Evicted?
- OOMKilled is about one container exceeding its own limit; the kernel kills it and the kubelet restarts it in place. Evicted is about the node running short; the kubelet removes whole pods, and their controller creates replacements elsewhere.
Sources: Kubernetes docs: Resource management for pods and containers ↗ · Kubernetes docs: Assign memory resources to containers and pods ↗ · Kubernetes docs: Node-pressure eviction ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.