Path 1 · lesson 2 of 4
Heap, stacks, Metaspace and native memory
Every memory area of a JVM process, which limit applies to each, which OutOfMemoryError points at which area, and why a container is killed while the heap looks fine.
What you will learn
- List the memory areas of a JVM process and the flag that limits each one
- Explain why process memory is always larger than -Xmx, and estimate by how much
- Map each OutOfMemoryError message to the area it points at and the first thing to check
- Tell a Java OutOfMemoryError from a kill by the container runtime
- Read jcmd VM.native_memory output and compare it with the resident set of the process
- Size the heap of a containerised service with MaxRAMPercentage and leave room for the rest
Before you start
The concept
-Xmx limits one thing: the Java heap, where objects live. A JVM process uses memory for much more than objects, and none of the rest is counted against -Xmx. The operating system and the container runtime see only the total.
| Area | What is in it | Limit | Default on JDK 21 |
|---|---|---|---|
| Java heap | Every object and array created with new. | -Xmx, or -XX:MaxRAMPercentage | 25% of the memory available to the JVM |
| Thread stacks | One stack per thread: a frame per method call with its local variables. | -Xss per thread, times the number of threads | 1024 KB per thread on Linux x64 |
| Metaspace | Class metadata: methods, bytecode, constant pools. Grows with the number of loaded classes. | -XX:MaxMetaspaceSize | Unlimited |
| Code cache | Machine code produced by the JIT compilers. | -XX:ReservedCodeCacheSize | 240 MB |
| Direct buffers | Memory behind ByteBuffer.allocateDirect, used by NIO, Netty, gRPC and many drivers. | -XX:MaxDirectMemorySize | Chosen by the JVM. On this JDK it equals the maximum heap size. |
| GC structures | Card tables, remembered sets, marking bitmaps. Grow with the heap. | None | A fraction of the heap |
| Everything else | The JVM's own data, symbols, the compiler's working memory, native libraries, memory allocated by JNI code, the C allocator's overhead. | None | Varies |
The lab program below was started with -Xmx256m. It holds 150 MB of live objects, 96 MB of direct buffers and 200 idle threads, and then prints its own numbers.
Program outputjava -Xmx256m -XX:NativeMemoryTracking=summary NativeMemory footprint
Real JVM outputpid 113218 heap: used 201 MB, committed 253 MB, max (-Xmx) 256 MB direct buffers held: 96 MB, live threads: 201 process resident set (what the container limit counts): VmRSS: 359468 kB
OutOfMemoryError, no unusual GC activity. The process, however, has a resident set of 359,468 kB, about 351 MB. That is 95 MB more than -Xmx, and it is the number a container limit is compared with. A dashboard that shows only heap usage would show a healthy service right up to the moment the container is killed.Why it exists
A JVM that runs out of heap tells you: it throws OutOfMemoryError, can write a heap dump, and leaves a stack trace. A process that exceeds its container memory limit tells you nothing. The kernel sends it SIGKILL. There is no exception, no heap dump, no last log line. Kubernetes reports the container as OOMKilled with exit code 137 and starts it again.
Program outputkubectl describe pod: a container killed for memory
Illustrative, not from a run State: Running
Started: Tue, 12 Nov 2024 03:12:41 +0000
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Mon, 11 Nov 2024 19:02:10 +0000
Finished: Tue, 12 Nov 2024 03:12:39 +0000
Restart Count: 4
Limits:
memory: 1GiReason: OOMKilled and Exit Code: 137 (128 + 9, SIGKILL) mean the kernel killed the process because the container went over its memory limit. This is not a Java error, and the application log will end without one. If the JVM itself had run out of heap you would find java.lang.OutOfMemoryError in the log, and the exit code would be whatever the application returned.The two failures need opposite fixes. Heap exhaustion with a healthy process is a heap problem: a leak or a heap that is too small, covered in Memory leaks and heap dumps and in the case The service that restarts every night. A container kill with a healthy heap usually means the heap is too large for the limit, or something outside the heap is growing. Raising -Xmx in that situation makes it worse.
How it works inside
Reserved, committed, resident
Three words describe how much memory an area has, and they are not the same number. Reserved is address space the JVM has claimed; it costs nothing until used. Committed is the part the JVM has asked the operating system to back with memory. Resident is what is actually in physical memory, which is what the container limit counts. A page that was committed and never touched is not resident.
Program outputjcmd <pid> VM.native_memory summary scale=MB (same process as above)
Real JVM outputNative Memory Tracking:
(Omitting categories weighting less than 1MB)
Total: reserved=1964MB, committed=438MB
malloc: 106MB #15064
mmap: reserved=1858MB, committed=332MB
- Java Heap (reserved=256MB, committed=253MB)
(mmap: reserved=256MB, committed=253MB, at peak)
...
- Thread (reserved=220MB, committed=22MB)
(threads #219)
(stack: reserved=219MB, committed=21MB, peak=21MB)
- Code (reserved=242MB, committed=7MB)
(mmap: reserved=242MB, committed=7MB, at peak)
- GC (reserved=42MB, committed=42MB)
(malloc=5MB #2190) (at peak)
(mmap: reserved=37MB, committed=37MB, at peak)
...
- Other (reserved=96MB, committed=96MB)
(malloc=96MB #98) (at peak)
...
- Shared class space (reserved=16MB, committed=13MB, readonly=0MB)
(mmap: reserved=16MB, committed=13MB, peak=13MB)
...
- Metaspace (reserved=64MB, committed=1MB)
(mmap: reserved=64MB, committed=1MB, at peak)committed column. Java Heap 253 MB: the whole -Xmx. Thread: 219 threads have 219 MB of stack reserved (1 MB each) but only 21 MB committed, because idle threads use a small part of their stack. Code: 242 MB reserved for the code cache, 7 MB used so far. GC: 42 MB of collector data for a 256 MB heap. Other: 96 MB, and this is where the direct buffers are. Total committed is 438 MB against a heap limit of 256 MB. The resident set printed by the program (351 MB) is lower than the committed total because some committed pages had not been touched yet; under load it moves towards the committed figure.The areas one by one
- Heap. Sized by
-Xmxor, without it, by-XX:MaxRAMPercentage(25% by default). The JVM grows it from the initial size as needed. Plan for the heap to reach its maximum: once heap memory has been committed, do not count on the JVM giving it back to the operating system soon. - Thread stacks. Each platform thread reserves
-Xss(1024 KB on Linux x64) and commits what its deepest call chain has used. 500 threads reserve 500 MB of address space and typically commit a few tens of megabytes. A single stack that runs out throwsStackOverflowError. Running out of threads or of memory for a new stack throwsOutOfMemoryError: unable to create native thread. - Metaspace. Class metadata, allocated per class loader and freed only when the loader is unloaded (see class loading). By default part of it, the compressed class space, is a separate region with 1 GB reserved;
-XX:MaxMetaspaceSizelimits the sum of both. A typical service settles at a stable value after startup. Growth after that is a loader leak or unbounded class generation. - Code cache. Holds compiled methods. 240 MB is reserved by default and it fills as the JIT compiles. It rarely needs attention; see JIT compilation for what happens when it is full.
- Direct buffers. Native memory owned by
ByteBufferobjects. The memory is released only after the smallByteBufferobject on the heap has been garbage collected, so a heap that is rarely collected can keep a lot of native memory alive. Network frameworks pool direct buffers; the pool size is theirs to configure, and-XX:MaxDirectMemorySizeis the hard ceiling. - Other native memory. Memory allocated by native libraries through JNI (compression, TLS, image and machine learning libraries), memory-mapped files, and the overhead of the C allocator. Native Memory Tracking does not see allocations made by third-party native code. If the resident set grows and NMT's total does not, the growth is here.
Which OutOfMemoryError means what
| Message | Area | First thing to check |
|---|---|---|
Java heap space | Heap | Is the live set growing (a leak) or is the heap simply too small? Heap after GC over time, then a histogram or heap dump. |
GC overhead limit exceeded | Heap | The collector is running almost constantly and freeing almost nothing. Same causes as Java heap space. Not all collectors raise it. |
Requested array size exceeds VM limit | None: a bug | Code asked for an array longer than the JVM allows. Look at the stack trace for a size computed from bad input. |
Metaspace | Metaspace | jcmd <pid> VM.classloader_stats: leaked loaders or generated classes. Raise MaxMetaspaceSize only if the count is stable. |
Compressed class space | Metaspace (class part) | Same as Metaspace. The region is limited by -XX:CompressedClassSpaceSize (1 GB by default). |
Cannot reserve N bytes of direct buffer memory (allocated: A, limit: L) | Direct buffers | Who allocates direct buffers and whether they are pooled. jvm.buffer.memory.used with tag id:direct. |
unable to create native thread | Threads / operating system | Thread count (jcmd <pid> Thread.print | grep -c '^"'), the process limit of the user or container, and unbounded thread creation. |
Out of swap space?, or an hs_err_pid file mentioning native memory allocation | Native memory of the whole machine or container | The JVM itself could not get native memory. Check container and host memory, then NMT. |
none: OOMKilled, exit code 137 | The whole process against the container limit | NMT total against the limit. The JVM was given no chance to report. |
Program outputOne program, five ways to run out
Real JVM output$ java -Xmx64m NativeMemory heap java.lang.OutOfMemoryError: Java heap space $ java -Xmx64m -XX:MaxDirectMemorySize=32m NativeMemory direct java.lang.OutOfMemoryError: Cannot reserve 1048576 bytes of direct buffer memory (allocated: 33554432, limit: 33554432) $ java -Xmx64m -XX:MaxMetaspaceSize=24m NativeMemory metaspace copies of NativeMemory$Generated loaded before the error: 3966 java.lang.OutOfMemoryError: Metaspace $ java -Xmx64m NativeMemory array java.lang.OutOfMemoryError: Requested array size exceeds VM limit $ java -Xmx64m -Xss256k NativeMemory stack frames on the stack when it overflowed: 1519 java.lang.StackOverflowError $ java -Xmx64m NativeMemory stack frames on the stack when it overflowed: 16646 java.lang.StackOverflowError
Metaspace was reached after 3,966 copies of one small class, each defined by its own class loader, against a limit of 24 MB. Requested array size exceeds VM limit appears at once with a 64 MB heap: the request (new long[Integer.MAX_VALUE]) can never succeed, whatever the heap size. The last two runs are not OutOfMemoryError: a stack overflow is a StackOverflowError. With -Xss256k the recursion reached 1,519 frames, with the default 1 MB stack 16,646. The second number varies from run to run, because compiled frames are smaller than interpreted ones.Program outputToo many threads
Illustrative, not from a runException in thread "http-nio-8080-exec-412" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached at java.base/java.lang.Thread.start0(Native Method) at java.base/java.lang.Thread.start(Thread.java:1553) at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
-Xss makes it worse.-Xmx, MaxRAMPercentage and containers
Without -Xmx the JVM sizes the heap from the memory available to it: -XX:MaxRAMPercentage, 25% by default. On Linux the JVM detects that it runs in a container (-XX:+UseContainerSupport, on by default) and uses the container's memory limit as the available memory, not the host's. The flag -XX:MaxRAM lets you see the calculation on any machine:
Program outputThe default heap for 512 MB of memory, and with MaxRAMPercentage=75
Real JVM output$ java -XX:MaxRAM=512m -XX:+PrintFlagsFinal -version | grep -E "MaxHeapSize|MaxRAM|UseContainerSupport"
size_t MaxHeapSize = 134217728 {product} {ergonomic}
uint64_t MaxRAM = 536870912 {pd product} {command line}
uintx MaxRAMFraction = 4 {product} {default}
double MaxRAMPercentage = 25.000000 {product} {default}
size_t SoftMaxHeapSize = 134217728 {manageable} {ergonomic}
bool UseContainerSupport = true {product} {default}
$ java -XX:MaxRAM=512m -XX:MaxRAMPercentage=75 -XX:+PrintFlagsFinal -version | grep -E "MaxHeapSize|MaxRAMPercentage"
size_t MaxHeapSize = 402653184 {product} {ergonomic}
double MaxRAMPercentage = 75.000000 {product} {command line}
size_t SoftMaxHeapSize = 402653184 {manageable} {ergonomic}-XX:MaxRAMPercentage=75 it is 402,653,184 bytes: 384 MB, leaving 128 MB for everything in the table above. {ergonomic} means the JVM computed the value, {command line} that you set it. UseContainerSupport is true by default. Run java -XX:+PrintFlagsFinal -version | grep MaxHeapSize inside your container to see what your service really gets.-Xmx768m
- A fixed number, independent of the container.
- Change the container limit and the heap stays the same: safe when the limit is raised, fatal when it is lowered below the heap plus the rest.
- Good when you have measured the live set and want the heap fixed.
-XX:MaxRAMPercentage=75
- A share of the container limit, computed at startup.
- Follows the limit: one setting for 512 MB and 4 GB pods.
- The non-heap part does not scale the same way. 25% of 512 MB is 128 MB, which is tight; 25% of 8 GB is 2 GB, mostly unused. Use a lower percentage for small containers.
Native Memory Tracking
NMT is the JVM's own accounting of its native memory by category. It must be switched on at startup with -XX:NativeMemoryTracking=summary (or detail); it cannot be enabled on a running process. The JDK documentation puts its cost at a 5 to 10% performance overhead, so decide per service whether it stays on. Then:
jcmd <pid> VM.native_memory summary scale=MBprints the table shown above.jcmd <pid> VM.native_memory baselineremembers the current state.- Some hours later,
jcmd <pid> VM.native_memory summary.diff scale=MBprints each category with its change since the baseline. The category that grew is your suspect. - Compare NMT's committed total with the resident set (
ps -o rss= -p <pid>, or the container's memory metric). If the resident set is far above NMT's total and rising, the memory is allocated by native code that NMT cannot see.
Architecture
Drawing diagram…
Flowchart. A container memory limit applies to the whole JVM process. The process consists of the Java heap and everything outside it: Metaspace, thread stacks, code cache, direct buffers, GC structures and native libraries. A full heap, Metaspace or direct buffer limit leads to a specific OutOfMemoryError. The process exceeding the container limit leads to a kill by the kernel with exit code 137.
flowchart TD L["Container memory limit, 1 GiB"] --> P["JVM process, resident memory"] P --> H["Java heap, limited by Xmx or MaxRAMPercentage"] P --> N["Everything outside the heap"] N --> M["Metaspace, class metadata"] N --> S["Thread stacks, Xss per thread"] N --> C["Code cache, compiled methods"] N --> D["Direct buffers, NIO and Netty"] N --> G["GC structures"] N --> O["Native libraries and allocator overhead"] H -->|"full"| E1["OutOfMemoryError Java heap space"] M -->|"full"| E2["OutOfMemoryError Metaspace"] D -->|"over its limit"| E3["OutOfMemoryError direct buffer memory"] P -->|"over the container limit"| K["Killed by the kernel, exit code 137"]
- Only the left branch is covered by
-Xmx. The limit at the top covers both branches. - Areas with their own limit fail with a Java error that names them. That is the good outcome: the process is alive, can log, and can write a heap dump.
- Areas with no limit, or with a limit above what the container can hold, fail as a kill of the whole process. Setting
-XX:MaxMetaspaceSizeand-XX:MaxDirectMemorySizeturns silent kills into named errors. - The heap has to be sized so that heap plus everything else stays under the container limit at peak, with margin. The rest is not a fixed percentage: measure it with NMT under load.
Step by step
Drawing diagram…
Sequence diagram. Application threads allocate objects and direct buffers. The JVM takes memory from the kernel for the heap and for native allocations. The heap stays under its limit, but the resident memory of the process reaches the container limit, and the kernel kills the process with SIGKILL. The kubelet sees exit code 137 with reason OOMKilled and starts a new container.
sequenceDiagram participant A as Application threads participant J as JVM participant K as Kernel cgroup participant O as Kubelet A->>J: Allocate objects, heap grows towards Xmx J->>K: Commit and touch more heap pages A->>J: Allocate direct buffers for uploads J->>K: Native allocation outside the heap Note over J: Heap is under Xmx, no Java error K->>K: Resident memory reaches the container limit K->>J: SIGKILL Note over J: No exception, no heap dump, no log line O->>K: Observes exit code 137, reason OOMKilled O->>J: Starts a new container Note over A: Warm-up starts again and requests in flight are lost
- The JVM enforces
-Xmxitself: when the heap is full it collects, and if that does not help it throwsOutOfMemoryError. Nothing in the JVM enforces the container limit for the process as a whole. - The kill is immediate.
-XX:+HeapDumpOnOutOfMemoryErrordoes nothing here, because noOutOfMemoryErrorwas ever thrown. - The evidence is outside the process: the pod's last state, the container runtime's events, and memory metrics of the container. Inside the process, the only preparation that helps is having NMT enabled before it happens.
In a real system
A document service runs in a container with a 1 GiB limit and -Xmx1g, on the reasoning that the heap should use what the container has. It serves uploads through a Netty-based HTTP client and server, with 200 worker threads. It is killed a few times a day under load, always with OOMKilled. Heap usage on the dashboard peaks around 70%.
| Area | At peak (example) | Where the number comes from |
|---|---|---|
| Java heap | up to 1,024 MB | -Xmx1g. The heap is allowed to commit all of it. |
| Metaspace | 130 MB | jvm.memory.used with tag id:Metaspace, plus the compressed class space |
| Thread stacks | 30 MB | NMT Thread, committed. About 250 threads. |
| Code cache | 70 MB | NMT Code, committed |
| GC structures | 60 MB | NMT GC |
| Direct buffers | 150 MB | jvm.buffer.memory.used with tag id:direct |
| Other | 40 MB | NMT Internal, Symbol, Arena Chunk and the rest |
| Total | about 1,500 MB | Against a limit of 1,024 MB |
The heap alone is allowed to take the entire limit. The process is killed as soon as heap plus the other 480 MB crosses 1 GiB, which happens when the heap has committed about 540 MB, long before it is 'full'. That is why the dashboard never shows the heap near 100%. The fix is to lower the heap, not to raise it: about 480 MB is needed outside the heap, so the heap gets at most 1,024 − 480 − a margin of 10%, roughly 450 MB. If the live set does not fit in that, the container needs a larger limit.
Then check each non-heap area for growth over days, with an NMT baseline and diff. Stable values mean the budget is right. A category that keeps growing is a leak of its own kind: Class and Metaspace (loaders), Thread (thread creation), Other (direct buffers), or resident memory beyond NMT's total (native libraries).
Java and Spring Boot
A Spring Boot 3.5 component that logs the non-heap areas next to the heap once a minute, using only JDK management APIs, and the Actuator queries for the same numbers. The Java file was compiled against stub Spring annotations here; it was not run inside Spring.
package com.example.documents;import java.lang.management.BufferPoolMXBean;import java.lang.management.ManagementFactory;import java.lang.management.MemoryPoolMXBean;import java.lang.management.MemoryType;import java.lang.management.MemoryUsage;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.scheduling.annotation.Scheduled;import org.springframework.stereotype.Component;/** Logs every memory area the JVM can report itself, so the heap is never looked at alone. */@Componentpublic class MemoryAreasReporter { private static final Logger log = LoggerFactory.getLogger(MemoryAreasReporter.class); private static final long MB = 1024 * 1024; @Scheduled(fixedRate = 60_000) public void report() { MemoryUsage heap = ManagementFactory.getMemoryMXBean().getHeapMemoryUsage(); long nonHeapCommitted = 0; StringBuilder pools = new StringBuilder(); for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) { if (pool.getType() == MemoryType.NON_HEAP) { nonHeapCommitted += pool.getUsage().getCommitted(); pools.append(' ').append(pool.getName()).append('=').append(pool.getUsage().getUsed() / MB).append("MB"); } } long direct = 0; for (BufferPoolMXBean buffers : ManagementFactory.getPlatformMXBeans(BufferPoolMXBean.class)) { if (buffers.getName().equals("direct")) { direct = buffers.getMemoryUsed(); } } int threads = ManagementFactory.getThreadMXBean().getThreadCount(); log.info("memory heapUsed={}MB heapCommitted={}MB heapMax={}MB nonHeapCommitted={}MB directBuffers={}MB threads={} pools:{}", heap.getUsed() / MB, heap.getCommitted() / MB, heap.getMax() / MB, nonHeapCommitted / MB, direct / MB, threads, pools); }}Why it is written this way
heapCommittedmatters more thanheapUsedfor the container: committed heap is what the process holds, whether or not objects are in it.- The non-heap pools are Metaspace, the compressed class space and the code cache segments. Their names appear in the log line, so a growing one is visible without a tool.
- The
directbuffer pool is the memory behindByteBuffer.allocateDirect. It is in no memory pool and is the area most often forgotten. - Thread stacks, GC structures and native libraries are not visible through these beans. The line is a lower bound for process memory; NMT and the container's own metric give the rest.
@Scheduledneeds@EnableSchedulingon a configuration class.
# Heap and non-heap, as Micrometer publishes them (Spring Boot Actuator, endpoint "metrics" exposed)curl -s "$APP/actuator/metrics/jvm.memory.committed?tag=area:heap"curl -s "$APP/actuator/metrics/jvm.memory.committed?tag=area:nonheap"curl -s "$APP/actuator/metrics/jvm.memory.used?tag=area:nonheap&tag=id:Metaspace"# Direct bufferscurl -s "$APP/actuator/metrics/jvm.buffer.memory.used?tag=id:direct"# Threadscurl -s "$APP/actuator/metrics/jvm.threads.live"# What the JVM cannot tell you: the process as the container sees itps -o rss= -p "$PID" # resident set in kBjcmd "$PID" VM.native_memory summary scale=MB # needs -XX:NativeMemoryTracking=summary at startupWhy it is written this way
- Meter names and the
areaandidtags are from Micrometer'sJvmMemoryMetricsandJvmThreadMetrics(1.15). Spring Boot publishes them underjvm.when Actuator is on the class path. - The metrics endpoint is not exposed over HTTP by default: add
metricstomanagement.endpoints.web.exposure.include. - Alert on container memory as a share of the limit, not on heap usage. Heap usage near 100% before a collection is normal; container memory near 100% is the last warning.
Wrong, then right
The same document service, deployed twice. The first deployment gives the heap the whole container. The second gives it a share and puts a ceiling on the areas that have none.
Incorrect implementation
apiVersion: apps/v1kind: Deploymentmetadata: name: document-servicespec: replicas: 3 selector: matchLabels: { app: document-service } template: metadata: labels: { app: document-service } spec: containers: - name: app image: registry.example.com/document-service:1.42.0 command: ["java"] args: - "-Xms1g" - "-Xmx1g" - "-jar" - "/app/document-service.jar" resources: requests: { memory: "1Gi", cpu: "1" } limits: { memory: "1Gi", cpu: "2" }Why it is written this way
-Xmx1gwithlimits.memory: 1Gileaves zero bytes for Metaspace, thread stacks, the code cache, direct buffers and GC structures.-Xms1gcommits the whole heap at startup, so the process is near the limit before the first request.- Nothing limits Metaspace or direct buffers, so growth there is found by the kernel and not by the JVM.
Why it fails
- The JVM is allowed to commit 1,024 MB for the heap alone. The service needs several hundred megabytes outside the heap, so the process crosses the 1 GiB limit while the heap is far from full, and the kernel kills it.
- Because no
OutOfMemoryErroris thrown,-XX:+HeapDumpOnOutOfMemoryErrorwould not fire, the log ends in the middle of normal work, and the dashboard shows heap usage at 60 to 70%. Every signal from inside the JVM says 'healthy'. - In production you see
OOMKilled, exit code 137 and a rising restart count, worst under load, when buffers and thread stacks are largest. A common wrong reaction is to raise-Xmx, which moves the kill earlier.
Corrected implementation
apiVersion: apps/v1kind: Deploymentmetadata: name: document-servicespec: replicas: 3 selector: matchLabels: { app: document-service } template: metadata: labels: { app: document-service } spec: containers: - name: app image: registry.example.com/document-service:1.42.0 command: ["java"] args: - "-XX:MaxRAMPercentage=50" - "-XX:MaxMetaspaceSize=192m" - "-XX:MaxDirectMemorySize=128m" - "-XX:NativeMemoryTracking=summary" - "-XX:+HeapDumpOnOutOfMemoryError" - "-XX:HeapDumpPath=/dumps" - "-XX:+ExitOnOutOfMemoryError" - "-jar" - "/app/document-service.jar" resources: requests: { memory: "1Gi", cpu: "1" } limits: { memory: "1Gi", cpu: "2" } volumeMounts: - { name: dumps, mountPath: /dumps } volumes: - name: dumps emptyDir: { sizeLimit: "2Gi" }Why it is written this way
MaxRAMPercentage=50gives a 512 MB heap in a 1 GiB container and leaves 512 MB for the rest. The share comes from the budget in the scenario above, where about 480 MB was needed outside the heap. If the live set does not fit in 512 MB, raise the container limit, not the percentage.MaxMetaspaceSizeandMaxDirectMemorySizeturn growth in those areas into anOutOfMemoryErrorthat names the area. The values are the measured peak plus headroom.NativeMemoryTracking=summarymakesjcmd VM.native_memoryavailable when it is needed. It cannot be switched on later.HeapDumpOnOutOfMemoryErrorwrites the dump to a volume that survives the container restart.ExitOnOutOfMemoryErrorthen ends the process, so a JVM that threwOutOfMemoryErrorin some thread does not limp on in an unknown state.- The memory request equals the limit, so the scheduler reserves what the process may really use.
Trade-offs
- A smaller heap means more frequent garbage collection for the same allocation rate, and less room for traffic peaks. If GC time rises too far, the answer is a larger container, not a larger share.
- Ceilings turn a kill into an error inside the JVM. That is easier to diagnose, but the limits are now yours to maintain: a new library that uses direct buffers can hit
MaxDirectMemorySizeafter an upgrade. - The JDK documents a 5 to 10% performance overhead for Native Memory Tracking. Measure it for your service and decide whether it stays on everywhere or only on one instance.
ExitOnOutOfMemoryErrorrestarts the pod on the firstOutOfMemoryError, including one caused by a single oversized request. Without it the JVM may survive such a request, at the risk of running on with broken state.- A heap dump of a 512 MB heap needs that much disk and pauses the process while it is written. The volume must be sized for it.
Configuration
The flags that size or limit a memory area. Defaults are from the JDK 21 java command reference, for Linux x64 where they depend on the platform.
-Xmx (-XX:MaxHeapSize)Default chosen at runtime: MaxRAMPercentage of the available memory
- What it does
- Maximum size of the Java heap. It limits nothing else.
- When to change it
- Set it, or MaxRAMPercentage, in every container. Heap plus all other areas must stay under the container limit at peak.
-XX:MaxRAMPercentageDefault 25
- What it does
- Maximum heap size as a percentage of the memory available to the JVM, which in a container is the container's memory limit.
- When to change it
- The default wastes most of a container. Values of 50 to 75 are common: lower for small containers, where the non-heap part is a larger share.
-XssDefault 1024 KB (Linux x64)
- What it does
- Stack size of each thread. Reserved per thread, committed as the stack is used.
- When to change it
- Raise it only for code that really recurses deeply. Lowering it saves little, since unused stack is not committed, and risks
StackOverflowError.
-XX:MaxMetaspaceSizeDefault unlimited
- What it does
- Maximum native memory for class metadata, including the compressed class space.
- When to change it
- Set it in containers to the stable value after warm-up plus headroom, so that a loader leak fails as
OutOfMemoryError: Metaspace.
-XX:MaxDirectMemorySizeDefault not set: the JVM chooses the limit (the maximum heap size on OpenJDK 21)
- What it does
- Maximum total size of NIO direct buffers.
- When to change it
- Set it explicitly. With the default, direct buffers may grow to the size of the heap again, which no container budget expects.
-XX:ReservedCodeCacheSizeDefault 240 MB (48 MB with -XX:-TieredCompilation)
- What it does
- Maximum size of the code cache for JIT-compiled code.
- When to change it
- Reserved, not committed: it costs what is actually compiled. Leave it unless NMT shows
Codenear the limit or the container is very small.
-XX:NativeMemoryTrackingDefault off
- What it does
summarytracks native memory by JVM subsystem,detailalso by call site. Read withjcmd <pid> VM.native_memory.- When to change it
- Enable
summarywhere you need to explain process memory. It has to be set at startup and costs some performance.
-XX:+UseContainerSupportDefault true (Linux)
- What it does
- The JVM reads the container's memory and CPU limits and sizes itself from them instead of from the host.
- When to change it
- Leave it on. Check the result with
-XX:+PrintFlagsFinalinside the container.
Hands-on exercise
Account for every megabyte of a JVM process
Run a program with a 256 MB heap limit, see that the process uses far more, and use Native Memory Tracking to say where the difference is. Then produce each out-of-memory error on purpose.
Steps · 0 of 6 done
What you should see
The heap is committed up to about 253 of 256 MB, while the resident set is around 350 MB and NMT reports about 438 MB committed. The difference of roughly 180 MB is mostly Other (96 MB, the direct buffers), GC (about 42 MB) and Thread (about 21 MB committed out of 219 MB reserved). The four error runs print Cannot reserve 1048576 bytes of direct buffer memory (allocated: 33554432, limit: 33554432), Java heap space, Metaspace and Requested array size exceeds VM limit. The default heap for 512 MB is 134217728 bytes (128 MB); with 75% it is 402653184 bytes (384 MB). Exact megabytes differ by a few on another machine.
What can go wrong
Setting -Xmx to the container limit
- Symptom
- The pod is
OOMKilledwith exit code 137 under load. Heap dashboards show 60 to 70% usage, and there is noOutOfMemoryErrorin the log. - Cause
- The heap may grow to the full limit, and everything outside the heap comes on top. The kernel kills the process when the sum crosses the limit.
- Fix
- Size the heap as limit minus the measured non-heap memory minus a margin. Use
-XX:MaxRAMPercentagein the 50 to 75 range and verify with NMT under load.
Running in a container with no heap setting at all
- Symptom
- A service in a 2 GiB container throws
OutOfMemoryError: Java heap spaceat about 500 MB of heap, with most of the container unused. - Cause
- The default
MaxRAMPercentageis 25, so the heap is a quarter of the container limit. - Fix
- Set
-XX:MaxRAMPercentageor-Xmxexplicitly, and checkMaxHeapSizewith-XX:+PrintFlagsFinalin the container.
Answering every OutOfMemoryError with a larger heap
- Symptom
OutOfMemoryError: Metaspace,direct buffer memoryorunable to create native threadkeeps returning after-Xmxwas raised, sometimes sooner than before.- Cause
- Those messages name areas outside the heap. A larger heap takes memory away from them inside the same container.
- Fix
- Read the message after the colon. It names the area: Metaspace means class loaders, direct buffer means NIO buffers, native thread means thread count or process limits.
Reading reserved memory as used memory
- Symptom
- NMT or
topshows gigabytes of virtual memory for a service with a 512 MB heap, and someone concludes the JVM is leaking. - Cause
- The JVM reserves address space generously: 1 GB for the compressed class space, 240 MB for the code cache, 1 MB per thread stack. Reserved memory is not backed by physical memory until it is committed and touched.
- Fix
- Look at
committedin NMT and at the resident set of the process. Ignorereservedand virtual size.
Trusting NMT to see everything
- Symptom
- The resident set climbs by hundreds of megabytes over days while NMT's committed total is flat.
- Cause
- NMT tracks the JVM's own allocations. Memory allocated by third-party native libraries through JNI, and overhead inside the C allocator, are outside it.
- Fix
- Compare resident set and NMT total regularly. If they diverge, look at native libraries in use (compression, TLS, image processing) and their own settings, with operating system tools.
How to troubleshoot
The container restarts and the application log has no error
Check
kubectl describe pod <pod> (Last State: Reason and Exit Code)
- How to read it
OOMKilledwith exit code 137 is a kill for exceeding the container memory limit, not a Java error. Go to process memory, not to the heap dump.
Need to know where process memory goes
Check
jcmd <pid> VM.native_memory summary scale=MB (requires -XX:NativeMemoryTracking=summary at startup)
- How to read it
- Compare
committedper category with your budget.Java Heapis-Xmx;Othercontains direct buffers;Threadscales with thread count;ClassandMetaspacewith loaded classes.
Process memory grows slowly over hours or days
Check
jcmd <pid> VM.native_memory baseline and later jcmd <pid> VM.native_memory summary.diff scale=MB
- How to read it
- The diff shows the change per category since the baseline. One growing category points at its cause. No growth in NMT while the resident set grows points at native code outside the JVM's accounting.
OutOfMemoryError: Cannot reserve ... bytes of direct buffer memory
Check
Micrometer `jvm.buffer.memory.used` and `jvm.buffer.count` with tag `id:direct`; the flag -XX:MaxDirectMemorySize
- How to read it
- The message prints the allocated amount and the limit. A count that rises with load and falls afterwards means the limit is too low for the peak. A count that only rises means buffers are not released.
OutOfMemoryError: unable to create native thread
Check
jcmd <pid> Thread.print | grep -c '^"' and Micrometer `jvm.threads.live`
- How to read it
- Thousands of threads mean something creates threads without a bound: an unbounded pool, a thread per request, a library's own pool per client. Group the thread names in the dump to find the owner.
Unsure what heap size the JVM chose in the container
Check
java -XX:+PrintFlagsFinal -version | grep -E 'MaxHeapSize|MaxRAMPercentage' (run inside the container)
- How to read it
MaxHeapSizein bytes is the effective-Xmx.{ergonomic}means the JVM computed it from the container limit; if it is a quarter of the limit, no heap flag reached the JVM.
Key takeaways
-Xmxlimits the heap only. The process also needs memory for Metaspace, thread stacks, the code cache, direct buffers, GC structures and native libraries.- A container limit applies to the whole process. Exceeding it is a kill by the kernel:
OOMKilled, exit code 137, no Java error and no heap dump. - The text after
OutOfMemoryError:names the area. OnlyJava heap spaceandGC overhead limit exceededare about the heap. - Without a heap flag the JVM takes 25% of the container limit. Set
-XX:MaxRAMPercentageor-Xmx, and leave room for the non-heap areas. - Give Metaspace and direct buffers explicit limits so that growth there fails as a named Java error.
- Read
committed, notreserved. Compare NMT's total with the resident set: a gap that grows is native memory the JVM does not track. - Native Memory Tracking must be enabled at startup. Decide before the incident.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 4
A service runs with -Xmx3g in a container with a 4 GiB limit. It is restarted about twice a day with OOMKilled, exit code 137. Heap after GC is steady at 1.6 GB. NMT shows: Java Heap committed 3,072 MB, Class 140 MB, Thread 310 MB (2,900 threads), Code 90 MB, GC 180 MB, Other 420 MB.
What is the best first change?
Scenario 2 of 4
After a library upgrade a service fails under load with java.lang.OutOfMemoryError: Cannot reserve 16777216 bytes of direct buffer memory (allocated: 520093696, limit: 536870912). It runs with -Xmx512m and no other memory flag. Heap usage is at 35%.
What does the message tell you?
Scenario 3 of 4
A team moves a service into a 2 GiB container and removes all JVM flags, 'because Java is container-aware now'. Under normal traffic it throws OutOfMemoryError: Java heap space. A heap histogram shows 480 MB of live objects, which was fine on the old host with 32 GB of RAM.
Why does it fail now?
Scenario 4 of 4
Over five days the resident set of a service grows from 1.1 GB to 1.9 GB. Heap after GC is flat at 400 MB with -Xmx768m. NMT summary.diff against a baseline from day one shows +12 MB in total. The service uses a native image-resizing library through JNI.
Where do you look next?
References
References: JDK 21: the java command (-Xmx, -Xss, MaxRAMPercentage, MaxMetaspaceSize, MaxDirectMemorySize, UseContainerSupport) ↗ · JDK 21 Troubleshooting Guide: Troubleshoot Memory Leaks (OutOfMemoryError messages) ↗ · JDK 21 HotSpot Virtual Machine Guide: Native Memory Tracking ↗ · JDK 21 Troubleshooting Guide: Diagnostic Tools (NMT memory categories, reserved and committed) ↗ · JDK 21 Garbage Collection Tuning Guide: Other Considerations (class metadata, compressed class space) ↗ · JDK 21: the jcmd command (VM.native_memory, VM.metaspace, Thread.print) ↗ · Kubernetes: Assign Memory Resources to Containers and Pods (exceeding a memory limit, OOMKilled) ↗ · Spring Boot 3.5 reference: Actuator metrics (JVM metrics, the metrics endpoint) ↗ · Report a problem