Skip to content

Path 1 · lesson 2 of 4

Intermediate28 minJava 21 (OpenJDK 21.0.13, HotSpot, G1) · Spring Boot 3.5 · Micrometer 1.15

Heap, stacks, Metaspace and native memory

Every memory area of a JVM process, which limit applies to each, which OutOfMemoryError points at which area, and why a container is killed while the heap looks fine.

What you will learn

  • List the memory areas of a JVM process and the flag that limits each one
  • Explain why process memory is always larger than -Xmx, and estimate by how much
  • Map each OutOfMemoryError message to the area it points at and the first thing to check
  • Tell a Java OutOfMemoryError from a kill by the container runtime
  • Read jcmd VM.native_memory output and compare it with the resident set of the process
  • Size the heap of a containerised service with MaxRAMPercentage and leave room for the rest

Before you start

The concept

-Xmx limits one thing: the Java heap, where objects live. A JVM process uses memory for much more than objects, and none of the rest is counted against -Xmx. The operating system and the container runtime see only the total.

AreaWhat is in itLimitDefault on JDK 21
Java heapEvery object and array created with new.-Xmx, or -XX:MaxRAMPercentage25% of the memory available to the JVM
Thread stacksOne stack per thread: a frame per method call with its local variables.-Xss per thread, times the number of threads1024 KB per thread on Linux x64
MetaspaceClass metadata: methods, bytecode, constant pools. Grows with the number of loaded classes.-XX:MaxMetaspaceSizeUnlimited
Code cacheMachine code produced by the JIT compilers.-XX:ReservedCodeCacheSize240 MB
Direct buffersMemory behind ByteBuffer.allocateDirect, used by NIO, Netty, gRPC and many drivers.-XX:MaxDirectMemorySizeChosen by the JVM. On this JDK it equals the maximum heap size.
GC structuresCard tables, remembered sets, marking bitmaps. Grow with the heap.NoneA fraction of the heap
Everything elseThe JVM's own data, symbols, the compiler's working memory, native libraries, memory allocated by JNI code, the C allocator's overhead.NoneVaries
Defaults from the JDK 21 java command reference. The direct buffer default was measured: with -Xmx64m and no other flag the error message reports limit: 67108864.

The lab program below was started with -Xmx256m. It holds 150 MB of live objects, 96 MB of direct buffers and 200 idle threads, and then prints its own numbers.

Program outputjava -Xmx256m -XX:NativeMemoryTracking=summary NativeMemory footprint

Real JVM output
pid 113218
heap: used 201 MB, committed 253 MB, max (-Xmx) 256 MB
direct buffers held: 96 MB, live threads: 201
process resident set (what the container limit counts): VmRSS: 359468 kB
The heap is at its limit of 256 MB and is not over it: no OutOfMemoryError, no unusual GC activity. The process, however, has a resident set of 359,468 kB, about 351 MB. That is 95 MB more than -Xmx, and it is the number a container limit is compared with. A dashboard that shows only heap usage would show a healthy service right up to the moment the container is killed.

Why it exists

A JVM that runs out of heap tells you: it throws OutOfMemoryError, can write a heap dump, and leaves a stack trace. A process that exceeds its container memory limit tells you nothing. The kernel sends it SIGKILL. There is no exception, no heap dump, no last log line. Kubernetes reports the container as OOMKilled with exit code 137 and starts it again.

Program outputkubectl describe pod: a container killed for memory

Illustrative, not from a run
    State:          Running
      Started:      Tue, 12 Nov 2024 03:12:41 +0000
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 11 Nov 2024 19:02:10 +0000
      Finished:     Tue, 12 Nov 2024 03:12:39 +0000
    Restart Count:  4
    Limits:
      memory:  1Gi
Reason: OOMKilled and Exit Code: 137 (128 + 9, SIGKILL) mean the kernel killed the process because the container went over its memory limit. This is not a Java error, and the application log will end without one. If the JVM itself had run out of heap you would find java.lang.OutOfMemoryError in the log, and the exit code would be whatever the application returned.

The two failures need opposite fixes. Heap exhaustion with a healthy process is a heap problem: a leak or a heap that is too small, covered in Memory leaks and heap dumps and in the case The service that restarts every night. A container kill with a healthy heap usually means the heap is too large for the limit, or something outside the heap is growing. Raising -Xmx in that situation makes it worse.

How it works inside

Reserved, committed, resident

Three words describe how much memory an area has, and they are not the same number. Reserved is address space the JVM has claimed; it costs nothing until used. Committed is the part the JVM has asked the operating system to back with memory. Resident is what is actually in physical memory, which is what the container limit counts. A page that was committed and never touched is not resident.

Program outputjcmd <pid> VM.native_memory summary scale=MB (same process as above)

Real JVM output
Native Memory Tracking:

(Omitting categories weighting less than 1MB)

Total: reserved=1964MB, committed=438MB
       malloc: 106MB #15064
       mmap:   reserved=1858MB, committed=332MB

-                 Java Heap (reserved=256MB, committed=253MB)
                            (mmap: reserved=256MB, committed=253MB, at peak)
...
-                    Thread (reserved=220MB, committed=22MB)
                            (threads #219)
                            (stack: reserved=219MB, committed=21MB, peak=21MB)

-                      Code (reserved=242MB, committed=7MB)
                            (mmap: reserved=242MB, committed=7MB, at peak)

-                        GC (reserved=42MB, committed=42MB)
                            (malloc=5MB #2190) (at peak)
                            (mmap: reserved=37MB, committed=37MB, at peak)
...
-                     Other (reserved=96MB, committed=96MB)
                            (malloc=96MB #98) (at peak)
...
-        Shared class space (reserved=16MB, committed=13MB, readonly=0MB)
                            (mmap: reserved=16MB, committed=13MB, peak=13MB)
...
-                 Metaspace (reserved=64MB, committed=1MB)
                            (mmap: reserved=64MB, committed=1MB, at peak)
Read the committed column. Java Heap 253 MB: the whole -Xmx. Thread: 219 threads have 219 MB of stack reserved (1 MB each) but only 21 MB committed, because idle threads use a small part of their stack. Code: 242 MB reserved for the code cache, 7 MB used so far. GC: 42 MB of collector data for a 256 MB heap. Other: 96 MB, and this is where the direct buffers are. Total committed is 438 MB against a heap limit of 256 MB. The resident set printed by the program (351 MB) is lower than the committed total because some committed pages had not been touched yet; under load it moves towards the committed figure.

The areas one by one

  • Heap. Sized by -Xmx or, without it, by -XX:MaxRAMPercentage (25% by default). The JVM grows it from the initial size as needed. Plan for the heap to reach its maximum: once heap memory has been committed, do not count on the JVM giving it back to the operating system soon.
  • Thread stacks. Each platform thread reserves -Xss (1024 KB on Linux x64) and commits what its deepest call chain has used. 500 threads reserve 500 MB of address space and typically commit a few tens of megabytes. A single stack that runs out throws StackOverflowError. Running out of threads or of memory for a new stack throws OutOfMemoryError: unable to create native thread.
  • Metaspace. Class metadata, allocated per class loader and freed only when the loader is unloaded (see class loading). By default part of it, the compressed class space, is a separate region with 1 GB reserved; -XX:MaxMetaspaceSize limits the sum of both. A typical service settles at a stable value after startup. Growth after that is a loader leak or unbounded class generation.
  • Code cache. Holds compiled methods. 240 MB is reserved by default and it fills as the JIT compiles. It rarely needs attention; see JIT compilation for what happens when it is full.
  • Direct buffers. Native memory owned by ByteBuffer objects. The memory is released only after the small ByteBuffer object on the heap has been garbage collected, so a heap that is rarely collected can keep a lot of native memory alive. Network frameworks pool direct buffers; the pool size is theirs to configure, and -XX:MaxDirectMemorySize is the hard ceiling.
  • Other native memory. Memory allocated by native libraries through JNI (compression, TLS, image and machine learning libraries), memory-mapped files, and the overhead of the C allocator. Native Memory Tracking does not see allocations made by third-party native code. If the resident set grows and NMT's total does not, the growth is here.

Which OutOfMemoryError means what

MessageAreaFirst thing to check
Java heap spaceHeapIs the live set growing (a leak) or is the heap simply too small? Heap after GC over time, then a histogram or heap dump.
GC overhead limit exceededHeapThe collector is running almost constantly and freeing almost nothing. Same causes as Java heap space. Not all collectors raise it.
Requested array size exceeds VM limitNone: a bugCode asked for an array longer than the JVM allows. Look at the stack trace for a size computed from bad input.
MetaspaceMetaspacejcmd <pid> VM.classloader_stats: leaked loaders or generated classes. Raise MaxMetaspaceSize only if the count is stable.
Compressed class spaceMetaspace (class part)Same as Metaspace. The region is limited by -XX:CompressedClassSpaceSize (1 GB by default).
Cannot reserve N bytes of direct buffer memory (allocated: A, limit: L)Direct buffersWho allocates direct buffers and whether they are pooled. jvm.buffer.memory.used with tag id:direct.
unable to create native threadThreads / operating systemThread count (jcmd <pid> Thread.print | grep -c '^"'), the process limit of the user or container, and unbounded thread creation.
Out of swap space?, or an hs_err_pid file mentioning native memory allocationNative memory of the whole machine or containerThe JVM itself could not get native memory. Check container and host memory, then NMT.
none: OOMKilled, exit code 137The whole process against the container limitNMT total against the limit. The JVM was given no chance to report.
Messages after the JDK 21 Troubleshooting Guide. The direct buffer and native thread messages are not listed there; the first is from the real run below, the second is the text JDK 21 uses.

Program outputOne program, five ways to run out

Real JVM output
$ java -Xmx64m NativeMemory heap
java.lang.OutOfMemoryError: Java heap space
$ java -Xmx64m -XX:MaxDirectMemorySize=32m NativeMemory direct
java.lang.OutOfMemoryError: Cannot reserve 1048576 bytes of direct buffer memory (allocated: 33554432, limit: 33554432)
$ java -Xmx64m -XX:MaxMetaspaceSize=24m NativeMemory metaspace
copies of NativeMemory$Generated loaded before the error: 3966
java.lang.OutOfMemoryError: Metaspace
$ java -Xmx64m NativeMemory array
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
$ java -Xmx64m -Xss256k NativeMemory stack
frames on the stack when it overflowed: 1519
java.lang.StackOverflowError
$ java -Xmx64m NativeMemory stack
frames on the stack when it overflowed: 16646
java.lang.StackOverflowError
Each run fills a different area and each message names it. The direct buffer message is the most helpful: it prints how much is allocated and the limit (32 MB here). Metaspace was reached after 3,966 copies of one small class, each defined by its own class loader, against a limit of 24 MB. Requested array size exceeds VM limit appears at once with a 64 MB heap: the request (new long[Integer.MAX_VALUE]) can never succeed, whatever the heap size. The last two runs are not OutOfMemoryError: a stack overflow is a StackOverflowError. With -Xss256k the recursion reached 1,519 frames, with the default 1 MB stack 16,646. The second number varies from run to run, because compiled frames are smaller than interpreted ones.

Program outputToo many threads

Illustrative, not from a run
Exception in thread "http-nio-8080-exec-412" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.lang.Thread.start0(Native Method)
	at java.base/java.lang.Thread.start(Thread.java:1553)
	at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
This one was not produced in the lab, to avoid exhausting a shared machine. Despite the name it is rarely about memory. The usual cause is an unbounded thread pool or a thread per request, running into the limit on processes for the user or the container. More heap does not help, and a larger -Xss makes it worse.

-Xmx, MaxRAMPercentage and containers

Without -Xmx the JVM sizes the heap from the memory available to it: -XX:MaxRAMPercentage, 25% by default. On Linux the JVM detects that it runs in a container (-XX:+UseContainerSupport, on by default) and uses the container's memory limit as the available memory, not the host's. The flag -XX:MaxRAM lets you see the calculation on any machine:

Program outputThe default heap for 512 MB of memory, and with MaxRAMPercentage=75

Real JVM output
$ java -XX:MaxRAM=512m -XX:+PrintFlagsFinal -version | grep -E "MaxHeapSize|MaxRAM|UseContainerSupport"
   size_t MaxHeapSize                              = 134217728                                 {product} {ergonomic}
 uint64_t MaxRAM                                   = 536870912                              {pd product} {command line}
    uintx MaxRAMFraction                           = 4                                         {product} {default}
   double MaxRAMPercentage                         = 25.000000                                 {product} {default}
   size_t SoftMaxHeapSize                          = 134217728                              {manageable} {ergonomic}
     bool UseContainerSupport                      = true                                      {product} {default}
$ java -XX:MaxRAM=512m -XX:MaxRAMPercentage=75 -XX:+PrintFlagsFinal -version | grep -E "MaxHeapSize|MaxRAMPercentage"
   size_t MaxHeapSize                              = 402653184                                 {product} {ergonomic}
   double MaxRAMPercentage                         = 75.000000                                 {product} {command line}
   size_t SoftMaxHeapSize                          = 402653184                              {manageable} {ergonomic}
With 512 MB available the default heap is 134,217,728 bytes: 128 MB, a quarter. That wastes most of a container. With -XX:MaxRAMPercentage=75 it is 402,653,184 bytes: 384 MB, leaving 128 MB for everything in the table above. {ergonomic} means the JVM computed the value, {command line} that you set it. UseContainerSupport is true by default. Run java -XX:+PrintFlagsFinal -version | grep MaxHeapSize inside your container to see what your service really gets.

-Xmx768m

  • A fixed number, independent of the container.
  • Change the container limit and the heap stays the same: safe when the limit is raised, fatal when it is lowered below the heap plus the rest.
  • Good when you have measured the live set and want the heap fixed.

-XX:MaxRAMPercentage=75

  • A share of the container limit, computed at startup.
  • Follows the limit: one setting for 512 MB and 4 GB pods.
  • The non-heap part does not scale the same way. 25% of 512 MB is 128 MB, which is tight; 25% of 8 GB is 2 GB, mostly unused. Use a lower percentage for small containers.

Native Memory Tracking

NMT is the JVM's own accounting of its native memory by category. It must be switched on at startup with -XX:NativeMemoryTracking=summary (or detail); it cannot be enabled on a running process. The JDK documentation puts its cost at a 5 to 10% performance overhead, so decide per service whether it stays on. Then:

  1. jcmd <pid> VM.native_memory summary scale=MB prints the table shown above.
  2. jcmd <pid> VM.native_memory baseline remembers the current state.
  3. Some hours later, jcmd <pid> VM.native_memory summary.diff scale=MB prints each category with its change since the baseline. The category that grew is your suspect.
  4. Compare NMT's committed total with the resident set (ps -o rss= -p <pid>, or the container's memory metric). If the resident set is far above NMT's total and rising, the memory is allocated by native code that NMT cannot see.

Architecture

Drawing diagram…

Flowchart. A container memory limit applies to the whole JVM process. The process consists of the Java heap and everything outside it: Metaspace, thread stacks, code cache, direct buffers, GC structures and native libraries. A full heap, Metaspace or direct buffer limit leads to a specific OutOfMemoryError. The process exceeding the container limit leads to a kill by the kernel with exit code 137.

flowchart TD
  L["Container memory limit, 1 GiB"] --> P["JVM process, resident memory"]
  P --> H["Java heap, limited by Xmx or MaxRAMPercentage"]
  P --> N["Everything outside the heap"]
  N --> M["Metaspace, class metadata"]
  N --> S["Thread stacks, Xss per thread"]
  N --> C["Code cache, compiled methods"]
  N --> D["Direct buffers, NIO and Netty"]
  N --> G["GC structures"]
  N --> O["Native libraries and allocator overhead"]
  H -->|"full"| E1["OutOfMemoryError Java heap space"]
  M -->|"full"| E2["OutOfMemoryError Metaspace"]
  D -->|"over its limit"| E3["OutOfMemoryError direct buffer memory"]
  P -->|"over the container limit"| K["Killed by the kernel, exit code 137"]
What a container limit has to cover, and which failure each area produces.
  • Only the left branch is covered by -Xmx. The limit at the top covers both branches.
  • Areas with their own limit fail with a Java error that names them. That is the good outcome: the process is alive, can log, and can write a heap dump.
  • Areas with no limit, or with a limit above what the container can hold, fail as a kill of the whole process. Setting -XX:MaxMetaspaceSize and -XX:MaxDirectMemorySize turns silent kills into named errors.
  • The heap has to be sized so that heap plus everything else stays under the container limit at peak, with margin. The rest is not a fixed percentage: measure it with NMT under load.

Step by step

Drawing diagram…

Sequence diagram. Application threads allocate objects and direct buffers. The JVM takes memory from the kernel for the heap and for native allocations. The heap stays under its limit, but the resident memory of the process reaches the container limit, and the kernel kills the process with SIGKILL. The kubelet sees exit code 137 with reason OOMKilled and starts a new container.

sequenceDiagram
  participant A as Application threads
  participant J as JVM
  participant K as Kernel cgroup
  participant O as Kubelet
  A->>J: Allocate objects, heap grows towards Xmx
  J->>K: Commit and touch more heap pages
  A->>J: Allocate direct buffers for uploads
  J->>K: Native allocation outside the heap
  Note over J: Heap is under Xmx, no Java error
  K->>K: Resident memory reaches the container limit
  K->>J: SIGKILL
  Note over J: No exception, no heap dump, no log line
  O->>K: Observes exit code 137, reason OOMKilled
  O->>J: Starts a new container
  Note over A: Warm-up starts again and requests in flight are lost
A container kill with a healthy heap: the JVM never learns that a limit existed for the memory outside the heap.
  1. The JVM enforces -Xmx itself: when the heap is full it collects, and if that does not help it throws OutOfMemoryError. Nothing in the JVM enforces the container limit for the process as a whole.
  2. The kill is immediate. -XX:+HeapDumpOnOutOfMemoryError does nothing here, because no OutOfMemoryError was ever thrown.
  3. The evidence is outside the process: the pod's last state, the container runtime's events, and memory metrics of the container. Inside the process, the only preparation that helps is having NMT enabled before it happens.

In a real system

A document service runs in a container with a 1 GiB limit and -Xmx1g, on the reasoning that the heap should use what the container has. It serves uploads through a Netty-based HTTP client and server, with 200 worker threads. It is killed a few times a day under load, always with OOMKilled. Heap usage on the dashboard peaks around 70%.

AreaAt peak (example)Where the number comes from
Java heapup to 1,024 MB-Xmx1g. The heap is allowed to commit all of it.
Metaspace130 MBjvm.memory.used with tag id:Metaspace, plus the compressed class space
Thread stacks30 MBNMT Thread, committed. About 250 threads.
Code cache70 MBNMT Code, committed
GC structures60 MBNMT GC
Direct buffers150 MBjvm.buffer.memory.used with tag id:direct
Other40 MBNMT Internal, Symbol, Arena Chunk and the rest
Totalabout 1,500 MBAgainst a limit of 1,024 MB
Example figures for one service, to show the method. Yours come from NMT and the buffer metrics.

The heap alone is allowed to take the entire limit. The process is killed as soon as heap plus the other 480 MB crosses 1 GiB, which happens when the heap has committed about 540 MB, long before it is 'full'. That is why the dashboard never shows the heap near 100%. The fix is to lower the heap, not to raise it: about 480 MB is needed outside the heap, so the heap gets at most 1,024 − 480 − a margin of 10%, roughly 450 MB. If the live set does not fit in that, the container needs a larger limit.

Then check each non-heap area for growth over days, with an NMT baseline and diff. Stable values mean the budget is right. A category that keeps growing is a leak of its own kind: Class and Metaspace (loaders), Thread (thread creation), Other (direct buffers), or resident memory beyond NMT's total (native libraries).

Java and Spring Boot

A Spring Boot 3.5 component that logs the non-heap areas next to the heap once a minute, using only JDK management APIs, and the Actuator queries for the same numbers. The Java file was compiled against stub Spring annotations here; it was not run inside Spring.

src/main/java/com/example/documents/MemoryAreasReporter.javaJava
package com.example.documents;import java.lang.management.BufferPoolMXBean;import java.lang.management.ManagementFactory;import java.lang.management.MemoryPoolMXBean;import java.lang.management.MemoryType;import java.lang.management.MemoryUsage;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.scheduling.annotation.Scheduled;import org.springframework.stereotype.Component;/** Logs every memory area the JVM can report itself, so the heap is never looked at alone. */@Componentpublic class MemoryAreasReporter {    private static final Logger log = LoggerFactory.getLogger(MemoryAreasReporter.class);    private static final long MB = 1024 * 1024;    @Scheduled(fixedRate = 60_000)    public void report() {        MemoryUsage heap = ManagementFactory.getMemoryMXBean().getHeapMemoryUsage();        long nonHeapCommitted = 0;        StringBuilder pools = new StringBuilder();        for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) {            if (pool.getType() == MemoryType.NON_HEAP) {                nonHeapCommitted += pool.getUsage().getCommitted();                pools.append(' ').append(pool.getName()).append('=').append(pool.getUsage().getUsed() / MB).append("MB");            }        }        long direct = 0;        for (BufferPoolMXBean buffers : ManagementFactory.getPlatformMXBeans(BufferPoolMXBean.class)) {            if (buffers.getName().equals("direct")) {                direct = buffers.getMemoryUsed();            }        }        int threads = ManagementFactory.getThreadMXBean().getThreadCount();        log.info("memory heapUsed={}MB heapCommitted={}MB heapMax={}MB nonHeapCommitted={}MB directBuffers={}MB threads={} pools:{}",                heap.getUsed() / MB, heap.getCommitted() / MB, heap.getMax() / MB,                nonHeapCommitted / MB, direct / MB, threads, pools);    }}

Why it is written this way

  • heapCommitted matters more than heapUsed for the container: committed heap is what the process holds, whether or not objects are in it.
  • The non-heap pools are Metaspace, the compressed class space and the code cache segments. Their names appear in the log line, so a growing one is visible without a tool.
  • The direct buffer pool is the memory behind ByteBuffer.allocateDirect. It is in no memory pool and is the area most often forgotten.
  • Thread stacks, GC structures and native libraries are not visible through these beans. The line is a lower bound for process memory; NMT and the container's own metric give the rest.
  • @Scheduled needs @EnableScheduling on a configuration class.

Wrong, then right

The same document service, deployed twice. The first deployment gives the heap the whole container. The second gives it a share and puts a ceiling on the areas that have none.

Incorrect implementation

k8s/deployment.yaml (heap equals the container limit)YAML
apiVersion: apps/v1kind: Deploymentmetadata:  name: document-servicespec:  replicas: 3  selector:    matchLabels: { app: document-service }  template:    metadata:      labels: { app: document-service }    spec:      containers:        - name: app          image: registry.example.com/document-service:1.42.0          command: ["java"]          args:            - "-Xms1g"            - "-Xmx1g"            - "-jar"            - "/app/document-service.jar"          resources:            requests: { memory: "1Gi", cpu: "1" }            limits: { memory: "1Gi", cpu: "2" }

Why it is written this way

  • -Xmx1g with limits.memory: 1Gi leaves zero bytes for Metaspace, thread stacks, the code cache, direct buffers and GC structures.
  • -Xms1g commits the whole heap at startup, so the process is near the limit before the first request.
  • Nothing limits Metaspace or direct buffers, so growth there is found by the kernel and not by the JVM.

Why it fails

  • The JVM is allowed to commit 1,024 MB for the heap alone. The service needs several hundred megabytes outside the heap, so the process crosses the 1 GiB limit while the heap is far from full, and the kernel kills it.
  • Because no OutOfMemoryError is thrown, -XX:+HeapDumpOnOutOfMemoryError would not fire, the log ends in the middle of normal work, and the dashboard shows heap usage at 60 to 70%. Every signal from inside the JVM says 'healthy'.
  • In production you see OOMKilled, exit code 137 and a rising restart count, worst under load, when buffers and thread stacks are largest. A common wrong reaction is to raise -Xmx, which moves the kill earlier.

Corrected implementation

k8s/deployment.yaml (heap as a share, ceilings on the rest)YAML
apiVersion: apps/v1kind: Deploymentmetadata:  name: document-servicespec:  replicas: 3  selector:    matchLabels: { app: document-service }  template:    metadata:      labels: { app: document-service }    spec:      containers:        - name: app          image: registry.example.com/document-service:1.42.0          command: ["java"]          args:            - "-XX:MaxRAMPercentage=50"            - "-XX:MaxMetaspaceSize=192m"            - "-XX:MaxDirectMemorySize=128m"            - "-XX:NativeMemoryTracking=summary"            - "-XX:+HeapDumpOnOutOfMemoryError"            - "-XX:HeapDumpPath=/dumps"            - "-XX:+ExitOnOutOfMemoryError"            - "-jar"            - "/app/document-service.jar"          resources:            requests: { memory: "1Gi", cpu: "1" }            limits: { memory: "1Gi", cpu: "2" }          volumeMounts:            - { name: dumps, mountPath: /dumps }      volumes:        - name: dumps          emptyDir: { sizeLimit: "2Gi" }

Why it is written this way

  • MaxRAMPercentage=50 gives a 512 MB heap in a 1 GiB container and leaves 512 MB for the rest. The share comes from the budget in the scenario above, where about 480 MB was needed outside the heap. If the live set does not fit in 512 MB, raise the container limit, not the percentage.
  • MaxMetaspaceSize and MaxDirectMemorySize turn growth in those areas into an OutOfMemoryError that names the area. The values are the measured peak plus headroom.
  • NativeMemoryTracking=summary makes jcmd VM.native_memory available when it is needed. It cannot be switched on later.
  • HeapDumpOnOutOfMemoryError writes the dump to a volume that survives the container restart. ExitOnOutOfMemoryError then ends the process, so a JVM that threw OutOfMemoryError in some thread does not limp on in an unknown state.
  • The memory request equals the limit, so the scheduler reserves what the process may really use.

Trade-offs

  • A smaller heap means more frequent garbage collection for the same allocation rate, and less room for traffic peaks. If GC time rises too far, the answer is a larger container, not a larger share.
  • Ceilings turn a kill into an error inside the JVM. That is easier to diagnose, but the limits are now yours to maintain: a new library that uses direct buffers can hit MaxDirectMemorySize after an upgrade.
  • The JDK documents a 5 to 10% performance overhead for Native Memory Tracking. Measure it for your service and decide whether it stays on everywhere or only on one instance.
  • ExitOnOutOfMemoryError restarts the pod on the first OutOfMemoryError, including one caused by a single oversized request. Without it the JVM may survive such a request, at the risk of running on with broken state.
  • A heap dump of a 512 MB heap needs that much disk and pauses the process while it is written. The volume must be sized for it.

Configuration

The flags that size or limit a memory area. Defaults are from the JDK 21 java command reference, for Linux x64 where they depend on the platform.

-Xmx (-XX:MaxHeapSize)Default chosen at runtime: MaxRAMPercentage of the available memory
What it does
Maximum size of the Java heap. It limits nothing else.
When to change it
Set it, or MaxRAMPercentage, in every container. Heap plus all other areas must stay under the container limit at peak.
-XX:MaxRAMPercentageDefault 25
What it does
Maximum heap size as a percentage of the memory available to the JVM, which in a container is the container's memory limit.
When to change it
The default wastes most of a container. Values of 50 to 75 are common: lower for small containers, where the non-heap part is a larger share.
-XssDefault 1024 KB (Linux x64)
What it does
Stack size of each thread. Reserved per thread, committed as the stack is used.
When to change it
Raise it only for code that really recurses deeply. Lowering it saves little, since unused stack is not committed, and risks StackOverflowError.
-XX:MaxMetaspaceSizeDefault unlimited
What it does
Maximum native memory for class metadata, including the compressed class space.
When to change it
Set it in containers to the stable value after warm-up plus headroom, so that a loader leak fails as OutOfMemoryError: Metaspace.
-XX:MaxDirectMemorySizeDefault not set: the JVM chooses the limit (the maximum heap size on OpenJDK 21)
What it does
Maximum total size of NIO direct buffers.
When to change it
Set it explicitly. With the default, direct buffers may grow to the size of the heap again, which no container budget expects.
-XX:ReservedCodeCacheSizeDefault 240 MB (48 MB with -XX:-TieredCompilation)
What it does
Maximum size of the code cache for JIT-compiled code.
When to change it
Reserved, not committed: it costs what is actually compiled. Leave it unless NMT shows Code near the limit or the container is very small.
-XX:NativeMemoryTrackingDefault off
What it does
summary tracks native memory by JVM subsystem, detail also by call site. Read with jcmd <pid> VM.native_memory.
When to change it
Enable summary where you need to explain process memory. It has to be set at startup and costs some performance.
-XX:+UseContainerSupportDefault true (Linux)
What it does
The JVM reads the container's memory and CPU limits and sizes itself from them instead of from the host.
When to change it
Leave it on. Check the result with -XX:+PrintFlagsFinal inside the container.

Hands-on exercise

Account for every megabyte of a JVM process

Run a program with a 256 MB heap limit, see that the process uses far more, and use Native Memory Tracking to say where the difference is. Then produce each out-of-memory error on purpose.

Steps · 0 of 6 done

What can go wrong

Setting -Xmx to the container limit
Symptom
The pod is OOMKilled with exit code 137 under load. Heap dashboards show 60 to 70% usage, and there is no OutOfMemoryError in the log.
Cause
The heap may grow to the full limit, and everything outside the heap comes on top. The kernel kills the process when the sum crosses the limit.
Fix
Size the heap as limit minus the measured non-heap memory minus a margin. Use -XX:MaxRAMPercentage in the 50 to 75 range and verify with NMT under load.
Running in a container with no heap setting at all
Symptom
A service in a 2 GiB container throws OutOfMemoryError: Java heap space at about 500 MB of heap, with most of the container unused.
Cause
The default MaxRAMPercentage is 25, so the heap is a quarter of the container limit.
Fix
Set -XX:MaxRAMPercentage or -Xmx explicitly, and check MaxHeapSize with -XX:+PrintFlagsFinal in the container.
Answering every OutOfMemoryError with a larger heap
Symptom
OutOfMemoryError: Metaspace, direct buffer memory or unable to create native thread keeps returning after -Xmx was raised, sometimes sooner than before.
Cause
Those messages name areas outside the heap. A larger heap takes memory away from them inside the same container.
Fix
Read the message after the colon. It names the area: Metaspace means class loaders, direct buffer means NIO buffers, native thread means thread count or process limits.
Reading reserved memory as used memory
Symptom
NMT or top shows gigabytes of virtual memory for a service with a 512 MB heap, and someone concludes the JVM is leaking.
Cause
The JVM reserves address space generously: 1 GB for the compressed class space, 240 MB for the code cache, 1 MB per thread stack. Reserved memory is not backed by physical memory until it is committed and touched.
Fix
Look at committed in NMT and at the resident set of the process. Ignore reserved and virtual size.
Trusting NMT to see everything
Symptom
The resident set climbs by hundreds of megabytes over days while NMT's committed total is flat.
Cause
NMT tracks the JVM's own allocations. Memory allocated by third-party native libraries through JNI, and overhead inside the C allocator, are outside it.
Fix
Compare resident set and NMT total regularly. If they diverge, look at native libraries in use (compression, TLS, image processing) and their own settings, with operating system tools.

How to troubleshoot

The container restarts and the application log has no error

Check

kubectl describe pod <pod>   (Last State: Reason and Exit Code)
How to read it
OOMKilled with exit code 137 is a kill for exceeding the container memory limit, not a Java error. Go to process memory, not to the heap dump.
Need to know where process memory goes

Check

jcmd <pid> VM.native_memory summary scale=MB   (requires -XX:NativeMemoryTracking=summary at startup)
How to read it
Compare committed per category with your budget. Java Heap is -Xmx; Other contains direct buffers; Thread scales with thread count; Class and Metaspace with loaded classes.
Process memory grows slowly over hours or days

Check

jcmd <pid> VM.native_memory baseline   and later   jcmd <pid> VM.native_memory summary.diff scale=MB
How to read it
The diff shows the change per category since the baseline. One growing category points at its cause. No growth in NMT while the resident set grows points at native code outside the JVM's accounting.
OutOfMemoryError: Cannot reserve ... bytes of direct buffer memory

Check

Micrometer `jvm.buffer.memory.used` and `jvm.buffer.count` with tag `id:direct`; the flag -XX:MaxDirectMemorySize
How to read it
The message prints the allocated amount and the limit. A count that rises with load and falls afterwards means the limit is too low for the peak. A count that only rises means buffers are not released.
OutOfMemoryError: unable to create native thread

Check

jcmd <pid> Thread.print | grep -c '^"'   and Micrometer `jvm.threads.live`
How to read it
Thousands of threads mean something creates threads without a bound: an unbounded pool, a thread per request, a library's own pool per client. Group the thread names in the dump to find the owner.
Unsure what heap size the JVM chose in the container

Check

java -XX:+PrintFlagsFinal -version | grep -E 'MaxHeapSize|MaxRAMPercentage'   (run inside the container)
How to read it
MaxHeapSize in bytes is the effective -Xmx. {ergonomic} means the JVM computed it from the container limit; if it is a quarter of the limit, no heap flag reached the JVM.

Key takeaways

  • -Xmx limits the heap only. The process also needs memory for Metaspace, thread stacks, the code cache, direct buffers, GC structures and native libraries.
  • A container limit applies to the whole process. Exceeding it is a kill by the kernel: OOMKilled, exit code 137, no Java error and no heap dump.
  • The text after OutOfMemoryError: names the area. Only Java heap space and GC overhead limit exceeded are about the heap.
  • Without a heap flag the JVM takes 25% of the container limit. Set -XX:MaxRAMPercentage or -Xmx, and leave room for the non-heap areas.
  • Give Metaspace and direct buffers explicit limits so that growth there fails as a named Java error.
  • Read committed, not reserved. Compare NMT's total with the resident set: a gap that grows is native memory the JVM does not track.
  • Native Memory Tracking must be enabled at startup. Decide before the incident.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 4

A service runs with -Xmx3g in a container with a 4 GiB limit. It is restarted about twice a day with OOMKilled, exit code 137. Heap after GC is steady at 1.6 GB. NMT shows: Java Heap committed 3,072 MB, Class 140 MB, Thread 310 MB (2,900 threads), Code 90 MB, GC 180 MB, Other 420 MB.

What is the best first change?

Scenario 2 of 4

After a library upgrade a service fails under load with java.lang.OutOfMemoryError: Cannot reserve 16777216 bytes of direct buffer memory (allocated: 520093696, limit: 536870912). It runs with -Xmx512m and no other memory flag. Heap usage is at 35%.

What does the message tell you?

Scenario 3 of 4

A team moves a service into a 2 GiB container and removes all JVM flags, 'because Java is container-aware now'. Under normal traffic it throws OutOfMemoryError: Java heap space. A heap histogram shows 480 MB of live objects, which was fine on the old host with 32 GB of RAM.

Why does it fail now?

Scenario 4 of 4

Over five days the resident set of a service grows from 1.1 GB to 1.9 GB. Heap after GC is flat at 400 MB with -Xmx768m. NMT summary.diff against a baseline from day one shows +12 MB in total. The service uses a native image-resizing library through JNI.

Where do you look next?

References

References: JDK 21: the java command (-Xmx, -Xss, MaxRAMPercentage, MaxMetaspaceSize, MaxDirectMemorySize, UseContainerSupport) ↗ · JDK 21 Troubleshooting Guide: Troubleshoot Memory Leaks (OutOfMemoryError messages) ↗ · JDK 21 HotSpot Virtual Machine Guide: Native Memory Tracking ↗ · JDK 21 Troubleshooting Guide: Diagnostic Tools (NMT memory categories, reserved and committed) ↗ · JDK 21 Garbage Collection Tuning Guide: Other Considerations (class metadata, compressed class space) ↗ · JDK 21: the jcmd command (VM.native_memory, VM.metaspace, Thread.print) ↗ · Kubernetes: Assign Memory Resources to Containers and Pods (exceeding a memory limit, OOMKilled) ↗ · Spring Boot 3.5 reference: Actuator metrics (JVM metrics, the metrics endpoint) ↗ · Report a problem