Skip to content
Breaks at Linux kernel · Your containerCan take the service downVery common

OOMKilled

The container used more memory than its limit, and the Linux kernel killed it. Exit code 137.

What you'll seeIllustrative output
$ kubectl get pods
NAME                     READY   STATUS      RESTARTS      AGE
worker-5c8f7d9b6-qm4zt   0/1     OOMKilled   3 (41s ago)   6m12s
$ kubectl describe pod worker-5c8f7d9b6-qm4zt
    State:          Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Restart Count:  3
    Limits:
      memory:  512Mi
    Requests:
      memory:  256Mi
$ kubectl top pod worker-5c8f7d9b6-qm4zt --containers
POD                      NAME     CPU(cores)   MEMORY(bytes)
worker-5c8f7d9b6-qm4zt   worker   212m         498Mi

Watch it happen

How OOMKilled unfolds

Step 3 · The failure

Restart, climb, killed again

The restarted container does the same work and climbs to the limit again. Each kill is followed by a longer back-off, so the pod drifts into CrashLoopBackOff. The kernel lights up each time: it is doing exactly what the limit told it to.

  • memory in use
  • kill
  • restart

Where it breaks: The Linux kernel does the killing, enforcing the memory limit you set on the container. Kubernetes only records the reason and restarts the container.

In short

What OOMKilled means

OOMKilled means the container tried to use more memory than its limit allows. The limit is not enforced by Kubernetes itself but by the Linux kernel, through the container's cgroup. When usage reaches the limit and nothing can be freed, the kernel's out-of-memory killer ends the process with SIGKILL. The process gets no warning and writes no error, which is why the application logs simply stop. Kubernetes records Reason OOMKilled with exit code 137 and the kubelet restarts the container, so repeated kills show up as CrashLoopBackOff. This is different from eviction, where the node as a whole is short of memory and the kubelet removes entire pods. The question that decides the fix is whether memory levels off below some value or climbs without end.

In order

Diagnose in 60 seconds

  1. 1
    kubectl describe pod <pod>
    Look for
    Last State: Reason OOMKilled, Exit Code 137. Note the memory limit listed under Limits.
    What it tells you
    Rules in: this container hit its own limit. If the reason is Error or Completed, it is not an OOM kill.
  2. 2
    kubectl get pod <pod> -o jsonpath='{.status.reason}{"\n"}'
    Look for
    An empty result, or the word Evicted.
    What it tells you
    Evicted means the node ran short of memory and the kubelet removed the whole pod. That is a node problem, not this container's limit.
  3. 3
    kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}'
    Look for
    The single word OOMKilled.
    What it tells you
    The same fact as describe, in a form you can script or run across many pods.
  4. 4
    kubectl top pod <pod> --containers
    Look for
    Current memory per container against the limit. Run it a few times as the container warms up. Needs metrics-server.
    What it tells you
    Rules in which container is growing, and how close to the limit it sits before each kill.
  5. 5
    kubectl logs <pod> --previous
    Look for
    What the app was doing just before the logs stop. There will be no error line: SIGKILL cannot be caught.
    What it tells you
    Points to the request, job or batch that was running when memory ran out.
  6. 6
    kubectl describe node <node>
    Look for
    The MemoryPressure condition, and SystemOOM or eviction events.
    What it tells you
    Rules in a node-level shortage affecting several pods, instead of one container's limit.

Most common first

Root causes

  1. 1The limit is lower than what the app really needs under load.

    Confirm
    Memory rises to roughly the same level every time and the kill comes at the limit, often during traffic peaks.
    Fix now
    Raise the limit above the measured peak, and raise the request with it.
    Prevent
    Size limits from load tests or from usage history, not from a guess.
  2. 2A memory leak: usage grows for as long as the container runs.

    Confirm
    The time between kills is long and regular, and usage never levels off whatever the load.
    Fix now
    Find and fix the leak. A higher limit only spaces the kills further apart.
    Prevent
    Alert on memory that keeps rising over hours, before it reaches the limit.
  3. 3A JVM (or another runtime) whose heap is sized as if it owned the whole container. A Java process also needs memory outside the heap: metaspace, thread stacks, direct buffers and other native memory.

    Confirm
    The maximum heap (-Xmx, or the share set by -XX:MaxRAMPercentage) plus that non-heap memory adds up to more than the container limit.
    Fix now
    Cap the heap at a share of the limit and leave the rest free, for example JAVA_TOOL_OPTIONS="-XX:MaxRAMPercentage=70". Never set -Xmx equal to the limit.
    Prevent
    Derive heap settings from the container's limit, and watch non-heap memory as well as heap in your JVM metrics.
  4. 4One large piece of work: a big upload, an export or a batch loaded into memory at once.

    Confirm
    Kills line up with particular requests or jobs, and the previous logs end in the middle of one.
    Fix now
    Stream or page the data, or move that work to a job with its own, larger limit.
    Prevent
    Put a size cap on inputs that are held in memory.
  5. 5Files written to a memory-backed volume count against the container's memory.

    Confirm
    The pod uses an emptyDir with medium: Memory, and the kills follow large writes to it.
    Fix now
    Set a sizeLimit on the volume, or use a disk-backed emptyDir.
    Prevent
    Treat memory-backed volumes as part of the memory budget.

The tempting wrong fix

Raise the memory limit

It works only if usage levels off. With a leak, a bigger limit just makes each crash take longer to arrive, and the larger container can push the whole node into memory pressure, where the kubelet starts evicting other pods. Check whether memory plateaus before changing the number.

Before / after

The most common fix: a limit that fits the measured peak

Before
resources:
  requests:
    memory: 256Mi
  limits:
    memory: 512Mi   # the app peaks at about 520Mi
After
resources:
  requests:
    memory: 768Mi   # equal to the limit: the scheduler reserves it
  limits:
    memory: 768Mi   # measured peak plus headroom

Setting the memory request equal to the limit means the node always has that memory set aside for the pod, so it is among the last candidates when the node runs short.

In the real world

Depends on version and provider

The kill is done by the Linux kernel, so details follow the node's setup. On nodes using cgroup v2, the kubelet normally has the kernel kill every process in the container together. On cgroup v1, or where the kubelet is configured for single-process kills, only one process may die: if it is not the main process, the container keeps running and no OOMKilled status appears. Node-level eviction thresholds also differ: the upstream default is memory.available below 100Mi on Linux, and managed providers set their own.

Seen in a real outage

More memory buys time, not a fix, when something keeps allocating. At PagerDuty in 2025, Kafka brokers ran out of heap one after another; raising the heap only delayed the next failure.

Break it: A producer for every request

Your turn

What would you do?

A container is OOMKilled about every six hours, whatever the traffic. Its memory graph is a steady upward line that never flattens. What does that tell you?

OOMKilled questions

What does OOMKilled mean in Kubernetes?
The container used more memory than its memory limit. The Linux kernel killed the process to enforce the limit, and Kubernetes recorded the reason as OOMKilled.
What is exit code 137?
It means the process was ended by SIGKILL: 128 plus signal number 9. With Reason OOMKilled it was the kernel's out-of-memory killer. Without it, something else sent the kill, such as the kubelet after a failed liveness probe.
What is the difference between OOMKilled and Evicted?
OOMKilled is about one container exceeding its own limit; the kernel kills it and the kubelet restarts it in place. Evicted is about the node running short; the kubelet removes whole pods, and their controller creates replacements elsewhere.

Sources: Kubernetes docs: Resource management for pods and containers ↗ · Kubernetes docs: Assign memory resources to containers and pods ↗ · Kubernetes docs: Node-pressure eviction ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play