Path 2 · lesson 3 of 4
Memory leaks and heap dumps
What a leak is on a garbage-collected heap, the code that causes one, what each OutOfMemoryError message points at, and how to find the leak from two histograms and a heap dump.
What you will learn
- Define a memory leak on a managed heap and tell it apart from a heap that is simply too small
- Recognise the usual causes in code: static collections, broken cache keys, listeners, ThreadLocals and class loaders
- Say which memory area each OutOfMemoryError message refers to and which tool applies
- Confirm a leak from the GC log and locate the growing class with two class histograms
- Capture a heap dump on OutOfMemoryError and on demand, knowing what each costs
- Use the dominator tree and the path to GC roots to find the reference that keeps the objects alive
Before you start
The concept
The collector frees every object the program can no longer reach. It follows that a Java program cannot leak memory the way a C program does, by losing the last pointer to a block. What it can do is the opposite: keep a reference to an object it will never use again. The collector has no way to know the object is useless. It is reachable, so it stays.
The word "grows" matters. A service that holds 3 GB of reference data from startup and stays at 3 GB has a large live set, not a leak. A service whose heap after collection climbs by 40 MB an hour, at constant traffic, has one. The first is a sizing question from the previous lesson. The second ends in java.lang.OutOfMemoryError however large the heap is made; a larger heap only moves the date.
| Heap too small | Leak | |
|---|---|---|
| Heap used after a full collection, over time | Flat, close to the limit | Rising without levelling off |
| Depends on | Load: more traffic, more live data | Time or number of requests, even at constant load |
| More heap | Fixes it | Delays it |
| A restart | Changes nothing | Resets the clock, which is why leaks show up as "it needs a restart every night" |
Why it exists
A leak does not fail when it is introduced. It fails days later, in a different place, usually at night: the request that finally cannot allocate is an innocent one, and its stack trace points at code that has nothing to do with the cause. Before that, the service gets slower, because the collector works harder and harder on a heap with less and less free space. This is the lab's leaking cache, run with a 64 MB heap.
Program outputA cache that never hits (java -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/leak.hprof HeapLeak leak)
Real JVM outputpid 4790, mode leak requests= 1,000 cache entries= 1,000 heap used= 11 MB of 64 MB requests= 2,000 cache entries= 2,000 heap used= 21 MB of 64 MB requests= 3,000 cache entries= 3,000 heap used= 31 MB of 64 MB requests= 4,000 cache entries= 4,000 heap used= 41 MB of 64 MB requests= 5,000 cache entries= 5,000 heap used= 50 MB of 64 MB requests= 6,000 cache entries= 6,000 heap used= 60 MB of 64 MB java.lang.OutOfMemoryError: Java heap space Dumping heap to /tmp/leak.hprof ... Heap dump file created [69295888 bytes in 0.024 secs] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"
GC logThe same run seen by the collector (-Xlog:gc)
Real JVM output[4.601s][info][gc] GC(4) Pause Young (Normal) (G1 Evacuation Pause) 33M->33M(64M) 6.615ms [4.610s][info][gc] GC(5) Pause Young (Concurrent Start) (G1 Evacuation Pause) 41M->41M(64M) 7.743ms ... [9.134s][info][gc] GC(12) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 60M->64M(64M) 2.061ms [9.145s][info][gc] GC(13) Pause Full (G1 Compaction Pause) 64M->60M(64M) 10.275ms [9.149s][info][gc] GC(14) Pause Young (Concurrent Start) (G1 Evacuation Pause) (Evacuation Failure) 62M->62M(64M) 2.264ms [9.149s][info][gc] GC(16) Concurrent Mark Cycle [9.155s][info][gc] GC(15) Pause Full (G1 Compaction Pause) 62M->62M(64M) 6.710ms [9.157s][info][gc] GC(16) Concurrent Mark Cycle 8.586ms [9.164s][info][gc] GC(17) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 63M->63M(64M) 2.570ms [9.170s][info][gc] GC(18) Pause Full (G1 Compaction Pause) 63M->63M(64M) 6.654ms [9.184s][info][gc] GC(19) Pause Full (G1 Compaction Pause) 63M->63M(64M) 13.908ms java.lang.OutOfMemoryError: Java heap space
33M->33M, 41M->41M: from early on, everything the collector looks at is still reachable. At the end come four full collections within 40 ms, the last two 63M->63M: the JVM tries everything it has, recovers nothing, and only then throws. A heap that is merely too small for its load shows full collections that do free memory, down to the same level each time.How it works inside
The usual causes
| Cause | What holds the objects | How it looks in the heap |
|---|---|---|
| A static or singleton collection that is only added to | A static field or a singleton bean, which lives as long as its class loader | One HashMap, ArrayList or ConcurrentHashMap with a retained size close to the whole leak |
A cache whose key has no equals/hashCode, or a mutable key | The cache map. Every lookup misses, so every request adds an entry | As many key objects as requests, where you expected as many as distinct customers |
| A cache with no bound | The cache map. Keys are correct but the key space is unbounded (URLs, user input, timestamps) | Entry count grows with the variety of traffic |
| Listeners and callbacks that are registered and never removed | The event source, through its listener list | Many instances of the listener class, each holding the screen, session or request it belonged to |
A ThreadLocal set on a pooled thread and never removed | The thread itself, which a pool keeps alive indefinitely | Values reachable from Thread objects through threadLocals, one per pool thread |
| A class loader that cannot be unloaded | Any object of a class from that loader that is still referenced from outside: a thread, a ThreadLocal, a driver registry, a shutdown hook | Several copies of the same classes, one per redeploy; Metaspace grows, not only the heap |
The ThreadLocal case follows from its documented behaviour: each thread holds an implicit reference to its copy of a thread-local variable as long as the thread is alive and the ThreadLocal instance is accessible. A request thread in a pool stays alive for the life of the process, so whatever a request stored and did not remove stays too, and the next request on that thread inherits it.
OutOfMemoryError is several different errors
The text after java.lang.OutOfMemoryError: names the memory area that ran out. Only some of them are about the Java heap, and a heap dump helps with only those.
| Message | What ran out | First tool |
|---|---|---|
Java heap space | The Java heap: an object could not be allocated even after full collections. A leak, or a heap too small for the load | GC log trend, class histograms, heap dump |
GC overhead limit exceeded | Still the heap. Thrown by the Parallel collector when more than 98% of the time goes to GC and less than 2% of the heap is recovered | Same as above |
Requested array size exceeds VM limit | Nothing: one array was requested that no heap could hold | The stack trace. A size computed from bad input, not a leak |
Metaspace | Native memory for class metadata, limited by -XX:MaxMetaspaceSize if set | jcmd <pid> VM.classloader_stats: look for many loaders of the same type |
Compressed class space | The fixed area for class metadata addressed by compressed class pointers | Same as Metaspace |
Cannot reserve N bytes of direct buffer memory | Off-heap memory used by direct ByteBuffers, limited by -XX:MaxDirectMemorySize | Find who allocates direct buffers; the heap histogram shows only small DirectByteBuffer objects |
... Out of swap space? | The operating system refused a native allocation by the JVM | Native memory tracking and the fatal error log, not a heap dump |
Program outputThree errors that a heap dump would not explain (MemoryLeaksHeapDumps.java)
Real JVM output$ java -XX:ActiveProcessorCount=2 -Xmx64m -XX:MaxMetaspaceSize=24m MemoryLeaksHeapDumps.java metaspace class loaders kept alive= 2,000 heap used=31 MB java.lang.OutOfMemoryError: Metaspace heap used at the time=15 MB of 64 MB $ java -XX:ActiveProcessorCount=2 -Xmx64m -XX:MaxDirectMemorySize=16m MemoryLeaksHeapDumps.java direct direct buffers= 4 MB heap used=16 MB direct buffers= 8 MB heap used=16 MB direct buffers=12 MB heap used=16 MB java.lang.OutOfMemoryError: Cannot reserve 1048576 bytes of direct buffer memory (allocated: 15736832, limit: 16777216) heap used at the time=4 MB of 64 MB $ java -XX:ActiveProcessorCount=2 -Xmx64m MemoryLeaksHeapDumps.java array java.lang.OutOfMemoryError: Requested array size exceeds VM limit heap used at the time=18 MB of 64 MB
Step 1: confirm it is a leak
Plot heap used after collection over hours or days: the after value of mixed or full collections in the GC log, or jvm.gc.live.data.size from Micrometer in a Spring Boot service. A sawtooth whose low points stay level is healthy. Low points that rise in a line, at constant traffic, are a leak. This costs nothing and should come before any dump.
Step 2: two class histograms
jcmd <pid> GC.class_histogram counts the live objects of every class and their total size. One histogram is hard to read: byte[] and String are at the top of every heap. Two histograms taken some time apart show what grew, and that is the leak.
HeapFirst histogram, after 3,000 requests (jcmd 4790 GC.class_histogram | head -12)
Real JVM output4790: num #instances #bytes class name (module) ------------------------------------------------------- 1: 12440 31168384 [B (java.base@21.0.13) 2: 9357 224568 java.lang.String (java.base@21.0.13) 3: 882 208448 [Ljdk.internal.vm.FillerElement; (java.base@21.0.13) 4: 1562 192880 java.lang.Class (java.base@21.0.13) 5: 5417 173344 java.util.HashMap$Node (java.base@21.0.13) 6: 1561 161456 [Ljava.lang.Object; (java.base@21.0.13) 7: 3000 72000 HeapLeak$LeakyKey 8: 217 56856 [C (java.base@21.0.13) 9: 275 51792 [Ljava.util.HashMap$Node; (java.base@21.0.13)
[B is byte[], [C is char[], [Ljava.lang.Object; is Object[]. Classes without a module in brackets are application classes. On its own this says that byte[] uses 31 MB and that there are exactly 3,000 HeapLeak$LeakyKey objects. A round number that equals the request count is already suspicious.HeapSecond histogram, after 6,000 requests (same command, a few seconds later)
Real JVM output4790: num #instances #bytes class name (module) ------------------------------------------------------- 1: 15424 61935760 [B (java.base@21.0.13) 2: 1006 411448 [Ljdk.internal.vm.FillerElement; (java.base@21.0.13) 3: 7287 233184 java.util.HashMap$Node (java.base@21.0.13) 4: 9341 224184 java.lang.String (java.base@21.0.13) 5: 1553 191808 java.lang.Class (java.base@21.0.13) 6: 1507 159928 [Ljava.lang.Object; (java.base@21.0.13) 7: 6000 144000 HeapLeak$LeakyKey 8: 273 59952 [Ljava.util.HashMap$Node; (java.base@21.0.13) 9: 217 56856 [C (java.base@21.0.13)
byte[]: 31.2 MB to 61.9 MB, with 2,984 more instances, so about 10 kB each: the size of one cached report. HeapLeak$LeakyKey: 3,000 to 6,000, exactly one per request. HashMap$Node grew as well, which says the keys sit in a hash map. String, Class and char[] did not move. The leak is a HashMap keyed by LeakyKey holding byte[] values. What the histogram cannot say is which map, or what refers to it.Step 3: a heap dump
A heap dump is a file with every object, its fields and its references, in HPROF format. It answers the question the histogram cannot: who holds these objects. There are two ways to get one.
- At the moment of failure.
-XX:+HeapDumpOnOutOfMemoryErrorwith-XX:HeapDumpPath=<file or directory>writes a dump when the firstOutOfMemoryErroris thrown. It is disabled by default and costs nothing until it fires. Without a path the file isjava_pid<pid>.hprofin the working directory, which in a container is often lost with the container: point the path at a mounted volume. - On demand.
jcmd <pid> GC.heap_dump <file>dumps a running process. By default it dumps live objects only and requests a full collection first;-allincludes unreachable objects. A Spring Boot service with Actuator can also serve one fromGET /actuator/heapdump, in HPROF format on HotSpot. That endpoint must never be reachable without authentication.
Program outputA dump on demand, and a dump on OutOfMemoryError followed by a clean exit
Real JVM output$ jcmd 370224 GC.heap_info 370224: garbage-first heap total 65536K, used 35180K [0x00000000fc000000, 0x0000000100000000) region size 1024K, 28 young (28672K), 3 survivors (3072K) Metaspace used 4159K, committed 4288K, reserved 1114112K class space used 391K, committed 448K, reserved 1048576K $ jcmd 370224 GC.heap_dump /tmp/leak-live.hprof 370224: Dumping heap to /tmp/leak-live.hprof ... Heap dump file created [39470933 bytes in 0.055 secs] ... java.lang.OutOfMemoryError: Java heap space Dumping heap to /tmp/x.hprof ... Heap dump file created [69430454 bytes in 0.023 secs] Terminating due to java.lang.OutOfMemoryError: Java heap space exit code 3
Pause Full (Heap Dump Initiated GC). The last four lines are from a run with -XX:+ExitOnOutOfMemoryError added: the dump is written first, then the JVM exits with code 3, so an orchestrator restarts it instead of leaving a half-dead process.Step 4: reading the dump in a heap analyser
Eclipse Memory Analyzer (MAT) is the usual tool. It was not run for this lesson, so what follows is its documented workflow in words. Two terms come first. The shallow heap of an object is the memory of that object alone. Its retained heap is everything that would be freed if it were collected: the object plus all objects reachable only through it. A HashMap has a shallow size of a few dozen bytes and can have a retained size of gigabytes.
- Open the dump (File > Open Heap Dump). MAT indexes it, which needs memory in proportion to the dump; analyse large dumps on a machine with plenty of RAM, not on a laptop with a default configuration.
- Run the Leak Suspects report. It looks for single objects or groups of objects that retain a suspiciously large share of the heap and names them. For a classic leak this is often the whole answer.
- Open the Dominator Tree. It lists objects by retained heap, largest first. An object dominates another if every path from the GC roots to the second passes through the first. The top entries are the objects that keep the most memory alive, and expanding one shows what it holds.
- Open the Histogram and compare it with what
jcmdshowed. From a class, list its objects with their incoming references to see who points at them. - Ask for the Path To GC Roots on one suspicious object, excluding weak and soft references. The result is the reference chain that keeps it alive, ending at a root such as a static field or a thread. The link in that chain that should not exist is the bug.
HeapWhat the dominator tree and the path to GC roots would show for this dump
Illustrative, not from a runDominator Tree
Class Name | Shallow Heap | Retained Heap | Percentage
-----------------------------------------------------------------------------------------------
java.util.HashMap @ 0xfc0a1b20 | 48 | 62,300,000 | 96.9%
java.util.HashMap$Node[8192] @ 0xfd3c4000 | 32,784 | 62,299,952 | 96.9%
java.util.HashMap$Node @ 0xfc1d2e08 | 32 | 10,312 | 0.0%
byte[10240] @ 0xfc1d2e28 | 10,256 | 10,256 | 0.0%
HeapLeak$LeakyKey @ 0xfc1d2de8 | 24 | 24 | 0.0%
... 5,999 more java.util.HashMap$Node
Path To GC Roots (excluding weak and soft references) for byte[10240] @ 0xfc1d2e28
byte[10240] @ 0xfc1d2e28
value java.util.HashMap$Node @ 0xfc1d2e08
[362] java.util.HashMap$Node[8192] @ 0xfd3c4000
table java.util.HashMap @ 0xfc0a1b20
CACHE class HeapLeak @ 0xfc0456a0 System ClassHashMap retains almost the whole heap. Read the path from the bottom: the map is the static field CACHE of class HeapLeak. A static field of a loaded class is a GC root, so nothing under it can be collected. The real JFR output below shows the same chain from an actual recording.A second route with no heap dump: JFR old object samples
Java Flight Recorder keeps a small sample of objects that were allocated and are still alive some time later, as jdk.OldObjectSample events. Dumping a recording with path-to-gc-roots=true adds the reference chain from each sampled object back to its root. It is a sample, not the whole heap, but it needs no multi-gigabyte file, and for a leak that dominates the heap the sample finds it.
JFROld object sample from the leaking run (recording started with -XX:StartFlightRecording=settings=profile)
Real JVM output$ jcmd 618933 JFR.dump name=1 path-to-gc-roots=true filename=/tmp/gcw/leak.jfr
618933:
Dumped recording "1", 302.3 kB written to:
/tmp/gcw/leak.jfr
$ jfr print --events jdk.OldObjectSample /tmp/gcw/leak.jfr
jdk.OldObjectSample {
startTime = 21:41:33.037 (2026-11-15)
duration = 23.4 ms
allocationTime = 21:41:26.601 (2026-11-15)
objectSize = 10.0 kB
objectAge = 6.44 s
lastKnownHeapUsage = 3.4 MB
object = [
byte[10240]
value : java.util.HashMap$Node
[362] : java.util.HashMap$Node[8192]
table : java.util.HashMap Size: 5000
CACHE : java.lang.Class Class Name: HeapLeak
]
arrayElements = 10240
root = {
description = "Thread Name: main"
system = "Threads"
type = "Stack Variable"
}
eventThread = "main" (javaThreadId = 1)
stackTrace = [
HeapLeak.lambda$render$0(Object) line: 30
java.util.HashMap.computeIfAbsent(Object, Function) line: 1228
HeapLeak.render(Object) line: 30
HeapLeak.main(String[]) line: 39
...
]
}object block is the reference chain, read downwards from the leaked object: a byte[10240] is the value of a HashMap$Node, which is element 362 of the node array, which is the table of a HashMap of size 5,000, which is the static field CACHE of class HeapLeak. The stackTrace is where the object was allocated: line 30 of HeapLeak.render, inside computeIfAbsent. Of the 22 sampled objects in this recording, 21 had this same chain. The recording is 302 kB; the heap dump of the same process was 69 MB.Architecture
Drawing diagram…
Flowchart of a reference chain. A static field CACHE is a GC root pointing to a HashMap, whose table of nodes holds one node per request. Each node references a LeakyKey for the same customer and report and a separate ten kilobyte byte array, so identical requests produce separate entries that all stay reachable.
flowchart LR R["GC root, static field CACHE of the service class"] --> M["HashMap"] M --> T["Node array, the table"] T --> N1["Node for request 1"] T --> N2["Node for request 2"] T --> N3["Node for request 6000"] N1 --> K1["LeakyKey customer 7, monthly"] N1 --> V1["byte array, 10 kB report"] N2 --> K2["LeakyKey customer 7, monthly"] N2 --> V2["byte array, 10 kB report"] N3 --> K3["LeakyKey customer 7, monthly"] N3 --> V3["byte array, 10 kB report"]
- The root is a static field. It lives as long as the class is loaded, which for an application class is the life of the process.
- Three nodes are shown with the same logical key, customer 7 and the monthly report. Because
LeakyKeydoes not defineequalsandhashCode, two keys with the same contents are different keys to the map, so each request gets its own node and its own 10 kB value. - A path to GC roots in a heap analyser, or the
objectblock of a JFR old object sample, is this picture read from right to left for one object. - The fix has to break the chain or stop it growing: make equal keys equal so that entries are reused, and bound the map so that even correct keys cannot grow it without limit.
Step by step
Drawing diagram…
Sequence diagram of a leak investigation. The engineer reads the heap trend, takes two class histograms some time apart and sees one class doubling. The JVM writes a heap dump on OutOfMemoryError, the engineer opens it in a heap analyser, finds one map retaining most of the heap and its path to a static field, then deploys a fix and repeats the histograms.
sequenceDiagram participant O as On-call engineer participant J as JVM of the service participant F as Files on disk participant A as Heap analyser O->>J: Read heap after GC over the last days J-->>O: Low points rise steadily at constant traffic O->>J: jcmd GC.class_histogram, first snapshot Note over O,J: Wait while the service handles more requests O->>J: jcmd GC.class_histogram, second snapshot J-->>O: byte arrays and one application class doubled J->>F: OutOfMemoryError, heap dump written to the dump path O->>A: Open the dump A-->>O: Dominator tree shows one HashMap retaining most of the heap O->>A: Path to GC roots for one entry A-->>O: Static field of the report service O->>J: Deploy the fix and repeat the two histograms
- The cheap steps come first. The heap trend and two histograms take minutes and often name the class. The dump, which is large, slow and sensitive, is for finding the holder.
- The dump on
OutOfMemoryErroris written by the JVM without anyone present, which is why the flag must be set before the incident, not during it. - The last step is the one most often skipped. Two histograms taken the same way after the fix, showing the suspect class flat, are the proof that the leak is gone and not merely slower.
In a real system
A reporting service renders a monthly statement per customer and caches the result. It is restarted by its orchestrator every night at about the same hour, with OutOfMemoryError: Java heap space in the last lines of the log. The team has doubled the heap twice; the restarts moved from every 12 hours to every 24 and then to every 48. That history alone is the diagnosis of a leak: more heap bought time in proportion and nothing else. The Investigate case on nightly restarts works through the same symptom.
- Trend.
jvm.gc.live.data.sizerises in a straight line between restarts while request rate follows its usual daily curve. A live set that tracks time and not load is a leak. - Histograms. Two
jcmd <pid> GC.class_histogramsnapshots an hour apart on one instance.byte[]and one application key class have grown by the same count; everything else is level. The key class has as many instances as requests served, where the team expected one per customer and report. - Dump. The next nightly failure writes a dump, because
-XX:+HeapDumpOnOutOfMemoryErrorand a dump path on a persistent volume were added after the first look. - Holder. The dominator tree shows one
HashMapretaining over 90% of the heap. The path to GC roots from one of its values ends at a static field of the report service. - Cause. The key class has two fields and no
equalsorhashCode. It was a record until someone turned it into a class to add a mutable field. - Fix and proof. The key becomes a record again and the cache gets a size bound. Two histograms after the deploy show 500 keys, then 500 keys.
Program outputAfter the fix: the same load with a record as the key (java -Xmx64m HeapLeak fixed)
Real JVM outputpid 4873, mode fixed requests= 1,000 cache entries= 500 heap used= 6 MB of 64 MB requests= 2,000 cache entries= 500 heap used= 6 MB of 64 MB requests= 3,000 cache entries= 500 heap used= 7 MB of 64 MB ... requests= 19,000 cache entries= 500 heap used= 7 MB of 64 MB requests= 20,000 cache entries= 500 heap used= 7 MB of 64 MB finished without running out of memory
Java and Spring Boot
What to have in place before a leak happens: the JVM flags that capture evidence at the moment of failure, a script for the two-histogram comparison, and the discipline for the leak that hides best, a ThreadLocal on a pooled thread. The flags and the script's commands were run on OpenJDK 21.0.13 here. TenantContext was compiled with JDK 21.
JAVA_TOOL_OPTIONS="-Xms2g -Xmx2g \ -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/dumps \ -XX:+ExitOnOutOfMemoryError \ -Xlog:gc*:file=/var/log/reports/gc.log:time,uptime,level,tags:filecount=5,filesize=20m"Why it is written this way
HeapDumpPathmay be a directory; the file is then namedjava_pid<pid>.hprofinside it./var/dumpsmust be a volume that survives the container and has at least as much free space as the heap.-XX:+ExitOnOutOfMemoryErrormakes the JVM exit after the firstOutOfMemoryError. A process that has failed an allocation is in an unknown state: some thread lost an operation halfway. In the run above the dump was written first and the exit code was 3.- The GC log gives the trend of heap after collection without any other tooling, and shows the full collections that free nothing just before the error.
#!/usr/bin/env bash# Two class histograms N seconds apart, then the classes that grew, largest growth in bytes first.# Usage: histogram-diff.sh <pid> [seconds] Run it on ONE instance: each histogram forces a full collection.set -euo pipefailpid="$1"; wait_s="${2:-300}"jcmd "$pid" GC.class_histogram > /tmp/histo-1.txtsleep "$wait_s"jcmd "$pid" GC.class_histogram > /tmp/histo-2.txt# Columns of a histogram row: rank, instances, bytes, class nameawk 'FNR > 3 && NF >= 4 { key = $4 } FNR > 3 && NF >= 4 && NR == FNR { inst[key] = $2; bytes[key] = $3; next } FNR > 3 && NF >= 4 && (key in inst) && $2 > inst[key] { printf "%12d bytes %10d instances %s\n", $3 - bytes[key], $2 - inst[key], key }' /tmp/histo-1.txt /tmp/histo-2.txt | sort -n -r | head -15Why it is written this way
- The output is sorted by growth in bytes. In a leak, the top rows are the leaked values and their keys, growing by the same number of instances.
- Wait long enough for the growth to stand out from normal churn: minutes for a fast leak, an hour for a slow one. Both snapshots count live objects only, so short-lived garbage does not appear.
- Applied to the two real histograms above, the first row is
[Bwith 30,767,376 more bytes in 2,984 more instances, andHeapLeak$LeakyKeyis two rows below with 3,000 more instances.
package com.example.reports;import java.util.function.Supplier;/** Per-request tenant data on a ThreadLocal, with the one rule that keeps it from leaking: remove in finally. */public final class TenantContext { public record Tenant(String id, byte[] brandingAssets) {} private static final ThreadLocal<Tenant> CURRENT = new ThreadLocal<>(); private TenantContext() {} public static Tenant current() { Tenant tenant = CURRENT.get(); if (tenant == null) throw new IllegalStateException("no tenant bound to this thread"); return tenant; } /** Binds the tenant for the duration of the work and always unbinds it, even when the work throws. */ public static <T> T callAs(Tenant tenant, Supplier<T> work) { CURRENT.set(tenant); try { return work.get(); } finally { CURRENT.remove(); } }}Why it is written this way
- Request threads come from a pool and are never discarded. A value set and not removed stays reachable from the
Threadobject for the life of the process: one stale tenant, with its branding assets, per pool thread. remove()infinallyis the whole fix.set(null)is not equivalent: it leaves an entry in the thread's map.- The same leak is also a correctness bug: the next request handled by that thread would see the previous tenant.
- There is no public
set: the only way to bind a tenant iscallAs, so the cleanup cannot be forgotten by a caller.
Wrong, then right
The report cache from the lab, as it would look in a Spring Boot service. Rendering a monthly statement takes about 200 ms, 500 customers ask for one repeatedly, and the result is about 10 kB. A map from (customer, report type) to the rendered bytes is the obvious optimisation.
Incorrect implementation
package com.example.reports;import java.util.Map;import java.util.concurrent.ConcurrentHashMap;import org.springframework.stereotype.Service;@Servicepublic class ReportService { /** Looks like a value object. It is not: without equals() and hashCode(), identity is the only equality. */ static final class ReportKey { final long customerId; final String reportType; ReportKey(long customerId, String reportType) { this.customerId = customerId; this.reportType = reportType; } } private final Map<ReportKey, byte[]> cache = new ConcurrentHashMap<>(); private final ReportRenderer renderer; public ReportService(ReportRenderer renderer) { this.renderer = renderer; } public byte[] monthlyStatement(long customerId) { ReportKey key = new ReportKey(customerId, "monthly"); // a new object on every call return cache.computeIfAbsent(key, k -> renderer.render(k.customerId, k.reportType)); }}Why it is written this way
ReportKeyinheritsequalsandhashCodefromObject: two keys are equal only if they are the same object.new ReportKey(7, "monthly")is never equal to an earliernew ReportKey(7, "monthly").- The service is a singleton bean, so
cachelives as long as the application context: effectively a static field. - Nothing ever removes an entry.
Why it fails
- Every call builds a new key that equals no existing key, so
computeIfAbsentnever finds an entry. The cache has a hit rate of zero: every request pays the 200 ms of rendering it was meant to save. - Every miss inserts a new entry of about 10 kB that is reachable from the singleton bean and therefore never collected. Heap use after collection grows by 10 MB per thousand requests. In the real run with a 64 MB heap the process died at 6,000 requests.
- On the way there the collector works harder for less: young collections that free nothing, then repeated full collections, so latency degrades before the failure. The final
OutOfMemoryErroris thrown in whichever thread allocates next, often in unrelated code, which is why the stack trace of the error rarely points at the leak. - Tests do not catch it. A unit test that asks for one report and gets the right bytes passes. The only visible signs are a cache hit metric at zero, if anyone publishes one, and the heap trend.
Corrected implementation
package com.example.reports;import java.util.Collections;import java.util.LinkedHashMap;import java.util.Map;import org.springframework.stereotype.Service;@Servicepublic class ReportService { /** A record gets equals() and hashCode() from its components: equal contents, equal keys. */ record ReportKey(long customerId, String reportType) {} private static final int MAX_ENTRIES = 2_000; // 2,000 x 10 kB = 20 MB at most private final Map<ReportKey, byte[]> cache = Collections.synchronizedMap( new LinkedHashMap<>(256, 0.75f, true) { @Override protected boolean removeEldestEntry(Map.Entry<ReportKey, byte[]> eldest) { return size() > MAX_ENTRIES; } }); private final ReportRenderer renderer; public ReportService(ReportRenderer renderer) { this.renderer = renderer; } public byte[] monthlyStatement(long customerId) { ReportKey key = new ReportKey(customerId, "monthly"); byte[] cached = cache.get(key); if (cached != null) return cached; byte[] rendered = renderer.render(key.customerId(), key.reportType()); // outside the map's lock cache.put(key, rendered); return rendered; } public int cachedReports() { return cache.size(); }}Why it is written this way
- The record is the fix for the leak: a second request for customer 7 finds the first one's entry. In the real run the cache settled at 500 entries and 7 MB.
- The bound is the fix for the next leak. Correct keys still grow without limit if the key space does, for example when a date range becomes part of the key. An access-ordered
LinkedHashMapwithremoveEldestEntryevicts the least recently used entry. - Rendering happens outside the map's lock, so a slow render does not block other lookups. Two simultaneous misses for the same key may both render; the second
putoverwrites the first with identical content. cachedReports()is there to be published as a gauge. An entry count that keeps rising is visible weeks before anOutOfMemoryError.ReportRendereris assumed to exist. Both variants were compiled on JDK 21 against a stand-in for the Spring annotation; the behaviour quoted comes from the lab programHeapLeak.java, which uses the same two key types.
Trade-offs
- A bounded cache evicts. With more than 2,000 active keys some requests render again, so the bound has to come from the memory you can spare and the hit rate you need, and both should be measured.
- Cached statements go stale. The leaking version never served a stale report because it never served a cached one. A working cache needs an expiry or an invalidation when the underlying data changes.
Collections.synchronizedMapputs one lock around every lookup. At high request rates that lock becomes the bottleneck; a cache library with size-based eviction avoids it at the price of a dependency.- A record's components are final. If the key needs to change after creation, it should not be a map key at all: a key whose hash changes while it is in a map is lost in it, which is the same leak by another route.
Configuration
The flags and commands used to capture evidence. Defaults are those of OpenJDK 21 as documented in the java and jcmd command references and printed by java -XX:+PrintFlagsFinal -version on 21.0.13.
-XX:+HeapDumpOnOutOfMemoryErrorDefault Disabled
- What it does
- Write an HPROF heap dump when a
java.lang.OutOfMemoryErroris thrown. - When to change it
- Enable it on every service. It costs nothing until it fires, and the dump at the moment of failure cannot be obtained afterwards.
-XX:HeapDumpPathDefault `java_pid<pid>.hprof` in the working directory
- What it does
- File or directory for the dump.
- When to change it
- Point it at a persistent volume with free space of at least the heap size. In a container the working directory usually disappears with the failed container.
-XX:+ExitOnOutOfMemoryErrorDefault Disabled
- What it does
- Exit the JVM when an
OutOfMemoryErroris first thrown. - When to change it
- Enable it for services managed by an orchestrator. A process that survives an
OutOfMemoryErroris unreliable; a restart is cleaner. The heap dump is still written first.
-XX:MaxMetaspaceSizeDefault Unlimited
- What it does
- Upper limit on native memory for class metadata.
- When to change it
- Setting a generous limit turns a class loader leak into a clear
OutOfMemoryError: Metaspaceinstead of slow growth of the process until the container is killed.
-XX:MaxDirectMemorySizeDefault 0, meaning the JVM chooses the limit itself
- What it does
- Upper limit on memory reserved by direct
ByteBuffers. - When to change it
- Set it explicitly for services that use direct buffers heavily (network frameworks), so that the container limit accounts for it.
jcmd <pid> GC.class_histogramDefault Live objects only (`-all` is false)
- What it does
- Instance count and bytes per class.
- When to change it
- Take two, some time apart, and compare. It forces a full collection and pauses the application: one instance, not in a loop.
jcmd <pid> GC.heap_dump <file>Default Live objects only (`-all` is false), uncompressed, does not overwrite an existing file
- What it does
- Write an HPROF dump of a running process.
- When to change it
- Use it when you cannot wait for the failure.
-gz=1compresses the file. Expect a pause that grows with the heap, and handle the file as sensitive data.
Hands-on exercise
From a rising heap to the reference that holds it
Watch a heap that grows between collections, take two histograms to find which classes grow, follow the path to the GC root to find what holds them, then apply the fix and confirm that the growth has stopped.
Steps · 0 of 7 done
A report cache in a 64 MB heap
Simulation · not a real JVMLet the requests run, take two histograms some time apart and compare them, then follow the path from a leaked object to its GC root. Switch the key type to see the fix.
What you should see
With the leaking key the heap after collection rises in a straight line and ends in an OutOfMemoryError. Between two histograms the byte arrays and the key class grow by the same number of instances, one per request, about 10 kB per entry; in the real run they went from 3,000 to 6,000 keys and from 31 MB to 62 MB of byte arrays, and the process died at 6,000 requests. The path to the GC root runs from the byte array through a map node and the map's table to a static cache field. With the record key the cache holds 500 entries however many requests run. A larger heap only delays the failure.
What can go wrong
Answering every OutOfMemoryError with more heap
- Symptom
- The interval between restarts doubles each time the heap is doubled, and the restarts never stop.
- Cause
- The live set grows with time, not with load. Any heap size is reached eventually.
- Fix
- Plot heap after collection against time at constant traffic. If it rises, find the growing class with two histograms before touching
-Xmx.
Taking a heap dump for an error that is not about the heap
- Symptom
- The dump shows a heap that is a quarter full. The error was
Metaspaceor a direct buffer message. - Cause
- The message names the area that ran out. Class metadata and direct buffers are native memory and barely appear in a heap dump.
- Fix
- Read the text after
OutOfMemoryError:first. For Metaspace usejcmd <pid> VM.classloader_stats; for native memory use native memory tracking.
No dump when the failure finally happens
- Symptom
- The service died overnight with
Java heap spaceand there is nothing to analyse except a stack trace in unrelated code. - Cause
-XX:+HeapDumpOnOutOfMemoryErrorwas not set, or the dump was written to a container file system that was deleted with the container.- Fix
- Set the flag and a
-XX:HeapDumpPathon a persistent volume as part of the standard JVM options, and check that the volume has room for a file the size of the heap.
Reading the histogram top to bottom
- Symptom
- An investigation concludes that "the problem is byte arrays and Strings" and stalls there.
- Cause
byte[],StringandObject[]lead every histogram. A single snapshot shows what is large, not what is growing or who holds it.- Fix
- Compare two snapshots and look for the application class that grows by the same count as the generic ones. Then use a dump or JFR old object samples to find the holder.
ThreadLocal values left on pooled threads
- Symptom
- Memory grows in steps as the request pool reaches its maximum size, then stays high; occasionally a request sees another request's context.
- Cause
- A value was set on a pool thread and not removed. The thread, kept alive by the pool, holds it indefinitely.
- Fix
- Always pair
setwithremove()in afinallyblock, or hide the pair behind one method asTenantContext.callAsdoes.
How to troubleshoot
Heap after collection rises steadily at constant traffic
Check
jcmd <pid> GC.class_histogram | head -15 (twice, some minutes apart)
- How to read it
- Classes whose instance count rose between the two snapshots are the leak. Look for an application class growing by the same number as
[Borjava.util.HashMap$Node.
java.lang.OutOfMemoryError: Java heap space in the log
Check
grep -E 'Pause Full|OutOfMemoryError' gc.log | tail -20 and ls -la /var/dumps/*.hprof
- How to read it
- Full collections that free almost nothing just before the error (
63M->63M) mean the heap was full of reachable objects. The.hproffile is the dump to open in a heap analyser; its size is about the heap in use.
You need to know what holds the leaked objects without a full dump
Check
jcmd <pid> JFR.dump path-to-gc-roots=true filename=/tmp/leak.jfr then jfr print --events jdk.OldObjectSample /tmp/leak.jfr
- How to read it
- Needs a flight recording that is already running. Each event shows a sampled long-lived object, the reference chain to its GC root in the
objectblock, and with the profile settings the stack trace of its allocation.
java.lang.OutOfMemoryError: Metaspace
Check
jcmd <pid> VM.classloader_stats
- How to read it
- One row per class loader with the number of classes it loaded. Hundreds of loaders of the same type, where you expect one, mean loaders are created and kept alive: a class loader leak.
A Spring Boot service where you have HTTP access but no shell
Check
GET /actuator/metrics/jvm.gc.live.data.size and, authenticated, GET /actuator/heapdump
- How to read it
- The first is the size of the old generation after the last collection: its trend is the leak test. The second returns an HPROF dump on HotSpot. It stops the application while it runs and exposes everything in memory, so it must be secured.
Key takeaways
- A leak on a managed heap is a set of objects that is reachable, useless and growing. The collector is working correctly; a reference is wrong.
- The test is the trend of heap after collection at constant load. Flat and high is a sizing question; rising is a leak, and more heap only postpones it.
- Read the message:
Java heap spaceis the heap,Metaspaceand direct buffer errors are not, and a heap dump helps only with the first. - Two class histograms some time apart show what grows. The application class growing in step with
byte[]orHashMap$Nodenames the leak. - Set
-XX:+HeapDumpOnOutOfMemoryErrorwith a persistent-XX:HeapDumpPathbefore you need it, and treat dumps as sensitive data. - In a heap analyser, the dominator tree shows what retains the memory and the path to GC roots shows why it cannot be collected. JFR old object samples give the same chain from a sample.
- Map keys need
equalsandhashCode(use records), caches need a bound, andThreadLocalvalues needremove()infinally.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 4
A service with -Xmx4g is restarted by its orchestrator about every 30 hours with OutOfMemoryError: Java heap space. After the heap was raised to 8 GB the restarts come about every 60 hours. Traffic has the same daily shape every day. Heap after full collection is 1.1 GB an hour after start and grows by about 130 MB per hour.
What is the right next step?
Scenario 2 of 4
Two histograms taken 10 minutes apart show: [B from 410,000 to 530,000 instances, com.shop.PriceKey from 400,000 to 520,000, java.util.concurrent.ConcurrentHashMap$Node from 405,000 to 525,000. The catalogue has 30,000 products and each price lookup uses new PriceKey(productId, currency) against a cache.
What is the most likely defect?
Scenario 3 of 4
A service fails with java.lang.OutOfMemoryError: Metaspace. -XX:MaxMetaspaceSize=256m is set. A heap dump taken on the error shows the heap 30% full. The service loads tenant-specific plugins and reloads them when a tenant changes its configuration, a few hundred times a day.
Where should the investigation go?
Scenario 4 of 4
A team needs a heap dump from a production instance with a 24 GB heap that is leaking slowly. The instance serves live traffic, the node has 20 GB of free disk, and the service handles card payments.
Which plan is sound?
References
References: Java SE 21 Troubleshooting Guide: Troubleshoot Memory Leaks (OutOfMemoryError messages, histograms, heap dumps, JFR) ↗ · The jcmd command, JDK 21 (GC.class_histogram, GC.heap_dump, GC.heap_info, JFR.dump) ↗ · The java command, JDK 21 (-XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath, -XX:MaxMetaspaceSize) ↗ · ThreadLocal (Java SE 21 API) ↗ · Eclipse Memory Analyzer help: Basic Tutorial (histogram, dominator tree, path to GC roots, leak suspects) ↗ · Eclipse Memory Analyzer help: Shallow vs. Retained Heap ↗ · Spring Boot 3.5 Actuator API: Heap Dump (heapdump) ↗ · Report a problem