Skip to content

Path 2 · lesson 3 of 4

Intermediate30 minJava 21 (OpenJDK 21.0.13) · Spring Boot 3.5 · Eclipse Memory Analyzer (described, not run here)

Memory leaks and heap dumps

What a leak is on a garbage-collected heap, the code that causes one, what each OutOfMemoryError message points at, and how to find the leak from two histograms and a heap dump.

What you will learn

  • Define a memory leak on a managed heap and tell it apart from a heap that is simply too small
  • Recognise the usual causes in code: static collections, broken cache keys, listeners, ThreadLocals and class loaders
  • Say which memory area each OutOfMemoryError message refers to and which tool applies
  • Confirm a leak from the GC log and locate the growing class with two class histograms
  • Capture a heap dump on OutOfMemoryError and on demand, knowing what each costs
  • Use the dominator tree and the path to GC roots to find the reference that keeps the objects alive

Before you start

The concept

The collector frees every object the program can no longer reach. It follows that a Java program cannot leak memory the way a C program does, by losing the last pointer to a block. What it can do is the opposite: keep a reference to an object it will never use again. The collector has no way to know the object is useless. It is reachable, so it stays.

The word "grows" matters. A service that holds 3 GB of reference data from startup and stays at 3 GB has a large live set, not a leak. A service whose heap after collection climbs by 40 MB an hour, at constant traffic, has one. The first is a sizing question from the previous lesson. The second ends in java.lang.OutOfMemoryError however large the heap is made; a larger heap only moves the date.

Heap too smallLeak
Heap used after a full collection, over timeFlat, close to the limitRising without levelling off
Depends onLoad: more traffic, more live dataTime or number of requests, even at constant load
More heapFixes itDelays it
A restartChanges nothingResets the clock, which is why leaks show up as "it needs a restart every night"
The same OutOfMemoryError, two different problems. The trend of the heap after collection separates them.

Why it exists

A leak does not fail when it is introduced. It fails days later, in a different place, usually at night: the request that finally cannot allocate is an innocent one, and its stack trace points at code that has nothing to do with the cause. Before that, the service gets slower, because the collector works harder and harder on a heap with less and less free space. This is the lab's leaking cache, run with a 64 MB heap.

Program outputA cache that never hits (java -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/leak.hprof HeapLeak leak)

Real JVM output
pid 4790, mode leak
requests=  1,000  cache entries=  1,000  heap used= 11 MB of 64 MB
requests=  2,000  cache entries=  2,000  heap used= 21 MB of 64 MB
requests=  3,000  cache entries=  3,000  heap used= 31 MB of 64 MB
requests=  4,000  cache entries=  4,000  heap used= 41 MB of 64 MB
requests=  5,000  cache entries=  5,000  heap used= 50 MB of 64 MB
requests=  6,000  cache entries=  6,000  heap used= 60 MB of 64 MB
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/leak.hprof ...
Heap dump file created [69295888 bytes in 0.024 secs]

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"
Only 500 different customers ever ask for a report, so a working cache would hold 500 entries. This one has as many entries as requests: every lookup misses and adds another 10 kB. Heap use rises by 10 MB per thousand requests in a straight line until nothing is left. The last line is worth noticing too: the JVM was so short of memory that even reporting the error failed.

GC logThe same run seen by the collector (-Xlog:gc)

Real JVM output
[4.601s][info][gc] GC(4) Pause Young (Normal) (G1 Evacuation Pause) 33M->33M(64M) 6.615ms
[4.610s][info][gc] GC(5) Pause Young (Concurrent Start) (G1 Evacuation Pause) 41M->41M(64M) 7.743ms
...
[9.134s][info][gc] GC(12) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 60M->64M(64M) 2.061ms
[9.145s][info][gc] GC(13) Pause Full (G1 Compaction Pause) 64M->60M(64M) 10.275ms
[9.149s][info][gc] GC(14) Pause Young (Concurrent Start) (G1 Evacuation Pause) (Evacuation Failure) 62M->62M(64M) 2.264ms
[9.149s][info][gc] GC(16) Concurrent Mark Cycle
[9.155s][info][gc] GC(15) Pause Full (G1 Compaction Pause) 62M->62M(64M) 6.710ms
[9.157s][info][gc] GC(16) Concurrent Mark Cycle 8.586ms
[9.164s][info][gc] GC(17) Pause Young (Normal) (G1 Evacuation Pause) (Evacuation Failure) 63M->63M(64M) 2.570ms
[9.170s][info][gc] GC(18) Pause Full (G1 Compaction Pause) 63M->63M(64M) 6.654ms
[9.184s][info][gc] GC(19) Pause Full (G1 Compaction Pause) 63M->63M(64M) 13.908ms
java.lang.OutOfMemoryError: Java heap space
The signature of a leak in a GC log is collections that free nothing. 33M->33M, 41M->41M: from early on, everything the collector looks at is still reachable. At the end come four full collections within 40 ms, the last two 63M->63M: the JVM tries everything it has, recovers nothing, and only then throws. A heap that is merely too small for its load shows full collections that do free memory, down to the same level each time.

How it works inside

The usual causes

CauseWhat holds the objectsHow it looks in the heap
A static or singleton collection that is only added toA static field or a singleton bean, which lives as long as its class loaderOne HashMap, ArrayList or ConcurrentHashMap with a retained size close to the whole leak
A cache whose key has no equals/hashCode, or a mutable keyThe cache map. Every lookup misses, so every request adds an entryAs many key objects as requests, where you expected as many as distinct customers
A cache with no boundThe cache map. Keys are correct but the key space is unbounded (URLs, user input, timestamps)Entry count grows with the variety of traffic
Listeners and callbacks that are registered and never removedThe event source, through its listener listMany instances of the listener class, each holding the screen, session or request it belonged to
A ThreadLocal set on a pooled thread and never removedThe thread itself, which a pool keeps alive indefinitelyValues reachable from Thread objects through threadLocals, one per pool thread
A class loader that cannot be unloadedAny object of a class from that loader that is still referenced from outside: a thread, a ThreadLocal, a driver registry, a shutdown hookSeveral copies of the same classes, one per redeploy; Metaspace grows, not only the heap
Six causes that account for most leaks in services. In every one, the fault is a reference that outlives the object's usefulness.

The ThreadLocal case follows from its documented behaviour: each thread holds an implicit reference to its copy of a thread-local variable as long as the thread is alive and the ThreadLocal instance is accessible. A request thread in a pool stays alive for the life of the process, so whatever a request stored and did not remove stays too, and the next request on that thread inherits it.

OutOfMemoryError is several different errors

The text after java.lang.OutOfMemoryError: names the memory area that ran out. Only some of them are about the Java heap, and a heap dump helps with only those.

MessageWhat ran outFirst tool
Java heap spaceThe Java heap: an object could not be allocated even after full collections. A leak, or a heap too small for the loadGC log trend, class histograms, heap dump
GC overhead limit exceededStill the heap. Thrown by the Parallel collector when more than 98% of the time goes to GC and less than 2% of the heap is recoveredSame as above
Requested array size exceeds VM limitNothing: one array was requested that no heap could holdThe stack trace. A size computed from bad input, not a leak
MetaspaceNative memory for class metadata, limited by -XX:MaxMetaspaceSize if setjcmd <pid> VM.classloader_stats: look for many loaders of the same type
Compressed class spaceThe fixed area for class metadata addressed by compressed class pointersSame as Metaspace
Cannot reserve N bytes of direct buffer memoryOff-heap memory used by direct ByteBuffers, limited by -XX:MaxDirectMemorySizeFind who allocates direct buffers; the heap histogram shows only small DirectByteBuffer objects
... Out of swap space?The operating system refused a native allocation by the JVMNative memory tracking and the fatal error log, not a heap dump
Messages as listed in the JDK 21 troubleshooting guide, plus the direct buffer message produced by the run below.

Program outputThree errors that a heap dump would not explain (MemoryLeaksHeapDumps.java)

Real JVM output
$ java -XX:ActiveProcessorCount=2 -Xmx64m -XX:MaxMetaspaceSize=24m MemoryLeaksHeapDumps.java metaspace
class loaders kept alive= 2,000  heap used=31 MB
java.lang.OutOfMemoryError: Metaspace
heap used at the time=15 MB of 64 MB

$ java -XX:ActiveProcessorCount=2 -Xmx64m -XX:MaxDirectMemorySize=16m MemoryLeaksHeapDumps.java direct
direct buffers= 4 MB  heap used=16 MB
direct buffers= 8 MB  heap used=16 MB
direct buffers=12 MB  heap used=16 MB
java.lang.OutOfMemoryError: Cannot reserve 1048576 bytes of direct buffer memory (allocated: 15736832, limit: 16777216)
heap used at the time=4 MB of 64 MB

$ java -XX:ActiveProcessorCount=2 -Xmx64m  MemoryLeaksHeapDumps.java array
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
heap used at the time=18 MB of 64 MB
In all three the heap is far from full when the error is thrown: 15 MB, 4 MB and 18 MB of 64 MB. The first is a class loader leak in miniature: each iteration creates a loader and a class and keeps both reachable, and Metaspace fills after a little over 2,000 of them. The second runs out of the direct buffer limit, and the message states both the limit and what was already allocated. The third is a single impossible request. Read the message before choosing the tool. The heap figures printed before the error include the compiler that launches a single-file program.

Step 1: confirm it is a leak

Plot heap used after collection over hours or days: the after value of mixed or full collections in the GC log, or jvm.gc.live.data.size from Micrometer in a Spring Boot service. A sawtooth whose low points stay level is healthy. Low points that rise in a line, at constant traffic, are a leak. This costs nothing and should come before any dump.

Step 2: two class histograms

jcmd <pid> GC.class_histogram counts the live objects of every class and their total size. One histogram is hard to read: byte[] and String are at the top of every heap. Two histograms taken some time apart show what grew, and that is the leak.

HeapFirst histogram, after 3,000 requests (jcmd 4790 GC.class_histogram | head -12)

Real JVM output
4790:
 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         12440       31168384  [B (java.base@21.0.13)
   2:          9357         224568  java.lang.String (java.base@21.0.13)
   3:           882         208448  [Ljdk.internal.vm.FillerElement; (java.base@21.0.13)
   4:          1562         192880  java.lang.Class (java.base@21.0.13)
   5:          5417         173344  java.util.HashMap$Node (java.base@21.0.13)
   6:          1561         161456  [Ljava.lang.Object; (java.base@21.0.13)
   7:          3000          72000  HeapLeak$LeakyKey
   8:           217          56856  [C (java.base@21.0.13)
   9:           275          51792  [Ljava.util.HashMap$Node; (java.base@21.0.13)
Columns: rank by bytes, number of live instances, total shallow bytes, class. [B is byte[], [C is char[], [Ljava.lang.Object; is Object[]. Classes without a module in brackets are application classes. On its own this says that byte[] uses 31 MB and that there are exactly 3,000 HeapLeak$LeakyKey objects. A round number that equals the request count is already suspicious.

HeapSecond histogram, after 6,000 requests (same command, a few seconds later)

Real JVM output
4790:
 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         15424       61935760  [B (java.base@21.0.13)
   2:          1006         411448  [Ljdk.internal.vm.FillerElement; (java.base@21.0.13)
   3:          7287         233184  java.util.HashMap$Node (java.base@21.0.13)
   4:          9341         224184  java.lang.String (java.base@21.0.13)
   5:          1553         191808  java.lang.Class (java.base@21.0.13)
   6:          1507         159928  [Ljava.lang.Object; (java.base@21.0.13)
   7:          6000         144000  HeapLeak$LeakyKey
   8:           273          59952  [Ljava.util.HashMap$Node; (java.base@21.0.13)
   9:           217          56856  [C (java.base@21.0.13)
Compare row by row. byte[]: 31.2 MB to 61.9 MB, with 2,984 more instances, so about 10 kB each: the size of one cached report. HeapLeak$LeakyKey: 3,000 to 6,000, exactly one per request. HashMap$Node grew as well, which says the keys sit in a hash map. String, Class and char[] did not move. The leak is a HashMap keyed by LeakyKey holding byte[] values. What the histogram cannot say is which map, or what refers to it.

Step 3: a heap dump

A heap dump is a file with every object, its fields and its references, in HPROF format. It answers the question the histogram cannot: who holds these objects. There are two ways to get one.

  • At the moment of failure. -XX:+HeapDumpOnOutOfMemoryError with -XX:HeapDumpPath=<file or directory> writes a dump when the first OutOfMemoryError is thrown. It is disabled by default and costs nothing until it fires. Without a path the file is java_pid<pid>.hprof in the working directory, which in a container is often lost with the container: point the path at a mounted volume.
  • On demand. jcmd <pid> GC.heap_dump <file> dumps a running process. By default it dumps live objects only and requests a full collection first; -all includes unreachable objects. A Spring Boot service with Actuator can also serve one from GET /actuator/heapdump, in HPROF format on HotSpot. That endpoint must never be reachable without authentication.

Program outputA dump on demand, and a dump on OutOfMemoryError followed by a clean exit

Real JVM output
$ jcmd 370224 GC.heap_info
370224:
 garbage-first heap   total 65536K, used 35180K [0x00000000fc000000, 0x0000000100000000)
  region size 1024K, 28 young (28672K), 3 survivors (3072K)
 Metaspace       used 4159K, committed 4288K, reserved 1114112K
  class space    used 391K, committed 448K, reserved 1048576K

$ jcmd 370224 GC.heap_dump /tmp/leak-live.hprof
370224:
Dumping heap to /tmp/leak-live.hprof ...
Heap dump file created [39470933 bytes in 0.055 secs]
...
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/x.hprof ...
Heap dump file created [69430454 bytes in 0.023 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space
exit code 3
The dump file is about as large as the used heap: 39 MB for 35 MB in use, 69 MB when the 64 MB heap was full. Plan disk space accordingly; a 16 GB heap writes a 16 GB file. The GC log of this run recorded the cost of the on-demand dump as Pause Full (Heap Dump Initiated GC). The last four lines are from a run with -XX:+ExitOnOutOfMemoryError added: the dump is written first, then the JVM exits with code 3, so an orchestrator restarts it instead of leaving a half-dead process.

Step 4: reading the dump in a heap analyser

Eclipse Memory Analyzer (MAT) is the usual tool. It was not run for this lesson, so what follows is its documented workflow in words. Two terms come first. The shallow heap of an object is the memory of that object alone. Its retained heap is everything that would be freed if it were collected: the object plus all objects reachable only through it. A HashMap has a shallow size of a few dozen bytes and can have a retained size of gigabytes.

  1. Open the dump (File > Open Heap Dump). MAT indexes it, which needs memory in proportion to the dump; analyse large dumps on a machine with plenty of RAM, not on a laptop with a default configuration.
  2. Run the Leak Suspects report. It looks for single objects or groups of objects that retain a suspiciously large share of the heap and names them. For a classic leak this is often the whole answer.
  3. Open the Dominator Tree. It lists objects by retained heap, largest first. An object dominates another if every path from the GC roots to the second passes through the first. The top entries are the objects that keep the most memory alive, and expanding one shows what it holds.
  4. Open the Histogram and compare it with what jcmd showed. From a class, list its objects with their incoming references to see who points at them.
  5. Ask for the Path To GC Roots on one suspicious object, excluding weak and soft references. The result is the reference chain that keeps it alive, ending at a root such as a static field or a thread. The link in that chain that should not exist is the bug.

HeapWhat the dominator tree and the path to GC roots would show for this dump

Illustrative, not from a run
Dominator Tree
Class Name                                         | Shallow Heap | Retained Heap | Percentage
-----------------------------------------------------------------------------------------------
java.util.HashMap @ 0xfc0a1b20                     |           48 |    62,300,000 |     96.9%
  java.util.HashMap$Node[8192] @ 0xfd3c4000        |       32,784 |    62,299,952 |     96.9%
    java.util.HashMap$Node @ 0xfc1d2e08            |           32 |        10,312 |      0.0%
      byte[10240] @ 0xfc1d2e28                     |       10,256 |        10,256 |      0.0%
      HeapLeak$LeakyKey @ 0xfc1d2de8               |           24 |            24 |      0.0%
    ... 5,999 more java.util.HashMap$Node

Path To GC Roots (excluding weak and soft references) for byte[10240] @ 0xfc1d2e28
byte[10240] @ 0xfc1d2e28
  value java.util.HashMap$Node @ 0xfc1d2e08
    [362] java.util.HashMap$Node[8192] @ 0xfd3c4000
      table java.util.HashMap @ 0xfc0a1b20
        CACHE class HeapLeak @ 0xfc0456a0  System Class
Written for this lesson in the style of MAT's views; the addresses and sizes are invented, and only their proportions follow the real run. Read the dominator tree from the top: one HashMap retains almost the whole heap. Read the path from the bottom: the map is the static field CACHE of class HeapLeak. A static field of a loaded class is a GC root, so nothing under it can be collected. The real JFR output below shows the same chain from an actual recording.

A second route with no heap dump: JFR old object samples

Java Flight Recorder keeps a small sample of objects that were allocated and are still alive some time later, as jdk.OldObjectSample events. Dumping a recording with path-to-gc-roots=true adds the reference chain from each sampled object back to its root. It is a sample, not the whole heap, but it needs no multi-gigabyte file, and for a leak that dominates the heap the sample finds it.

JFROld object sample from the leaking run (recording started with -XX:StartFlightRecording=settings=profile)

Real JVM output
$ jcmd 618933 JFR.dump name=1 path-to-gc-roots=true filename=/tmp/gcw/leak.jfr
618933:
Dumped recording "1", 302.3 kB written to:

/tmp/gcw/leak.jfr

$ jfr print --events jdk.OldObjectSample /tmp/gcw/leak.jfr
jdk.OldObjectSample {
  startTime = 21:41:33.037 (2026-11-15)
  duration = 23.4 ms
  allocationTime = 21:41:26.601 (2026-11-15)
  objectSize = 10.0 kB
  objectAge = 6.44 s
  lastKnownHeapUsage = 3.4 MB
  object =  [
    byte[10240]
    value : java.util.HashMap$Node
    [362] : java.util.HashMap$Node[8192]
    table : java.util.HashMap Size: 5000
    CACHE : java.lang.Class Class Name: HeapLeak
  ]
  arrayElements = 10240
  root = {
    description = "Thread Name: main"
    system = "Threads"
    type = "Stack Variable"
  }
  eventThread = "main" (javaThreadId = 1)
  stackTrace = [
    HeapLeak.lambda$render$0(Object) line: 30
    java.util.HashMap.computeIfAbsent(Object, Function) line: 1228
    HeapLeak.render(Object) line: 30
    HeapLeak.main(String[]) line: 39
    ...
  ]
}
The object block is the reference chain, read downwards from the leaked object: a byte[10240] is the value of a HashMap$Node, which is element 362 of the node array, which is the table of a HashMap of size 5,000, which is the static field CACHE of class HeapLeak. The stackTrace is where the object was allocated: line 30 of HeapLeak.render, inside computeIfAbsent. Of the 22 sampled objects in this recording, 21 had this same chain. The recording is 302 kB; the heap dump of the same process was 69 MB.

Architecture

Drawing diagram…

Flowchart of a reference chain. A static field CACHE is a GC root pointing to a HashMap, whose table of nodes holds one node per request. Each node references a LeakyKey for the same customer and report and a separate ten kilobyte byte array, so identical requests produce separate entries that all stay reachable.

flowchart LR
  R["GC root, static field CACHE of the service class"] --> M["HashMap"]
  M --> T["Node array, the table"]
  T --> N1["Node for request 1"]
  T --> N2["Node for request 2"]
  T --> N3["Node for request 6000"]
  N1 --> K1["LeakyKey customer 7, monthly"]
  N1 --> V1["byte array, 10 kB report"]
  N2 --> K2["LeakyKey customer 7, monthly"]
  N2 --> V2["byte array, 10 kB report"]
  N3 --> K3["LeakyKey customer 7, monthly"]
  N3 --> V3["byte array, 10 kB report"]
Why nothing in the cache can be collected: every report is reachable from a static field through the map.
  • The root is a static field. It lives as long as the class is loaded, which for an application class is the life of the process.
  • Three nodes are shown with the same logical key, customer 7 and the monthly report. Because LeakyKey does not define equals and hashCode, two keys with the same contents are different keys to the map, so each request gets its own node and its own 10 kB value.
  • A path to GC roots in a heap analyser, or the object block of a JFR old object sample, is this picture read from right to left for one object.
  • The fix has to break the chain or stop it growing: make equal keys equal so that entries are reused, and bound the map so that even correct keys cannot grow it without limit.

Step by step

Drawing diagram…

Sequence diagram of a leak investigation. The engineer reads the heap trend, takes two class histograms some time apart and sees one class doubling. The JVM writes a heap dump on OutOfMemoryError, the engineer opens it in a heap analyser, finds one map retaining most of the heap and its path to a static field, then deploys a fix and repeats the histograms.

sequenceDiagram
  participant O as On-call engineer
  participant J as JVM of the service
  participant F as Files on disk
  participant A as Heap analyser
  O->>J: Read heap after GC over the last days
  J-->>O: Low points rise steadily at constant traffic
  O->>J: jcmd GC.class_histogram, first snapshot
  Note over O,J: Wait while the service handles more requests
  O->>J: jcmd GC.class_histogram, second snapshot
  J-->>O: byte arrays and one application class doubled
  J->>F: OutOfMemoryError, heap dump written to the dump path
  O->>A: Open the dump
  A-->>O: Dominator tree shows one HashMap retaining most of the heap
  O->>A: Path to GC roots for one entry
  A-->>O: Static field of the report service
  O->>J: Deploy the fix and repeat the two histograms
A leak investigation from the first suspicion to the verified fix.
  1. The cheap steps come first. The heap trend and two histograms take minutes and often name the class. The dump, which is large, slow and sensitive, is for finding the holder.
  2. The dump on OutOfMemoryError is written by the JVM without anyone present, which is why the flag must be set before the incident, not during it.
  3. The last step is the one most often skipped. Two histograms taken the same way after the fix, showing the suspect class flat, are the proof that the leak is gone and not merely slower.

In a real system

A reporting service renders a monthly statement per customer and caches the result. It is restarted by its orchestrator every night at about the same hour, with OutOfMemoryError: Java heap space in the last lines of the log. The team has doubled the heap twice; the restarts moved from every 12 hours to every 24 and then to every 48. That history alone is the diagnosis of a leak: more heap bought time in proportion and nothing else. The Investigate case on nightly restarts works through the same symptom.

  1. Trend. jvm.gc.live.data.size rises in a straight line between restarts while request rate follows its usual daily curve. A live set that tracks time and not load is a leak.
  2. Histograms. Two jcmd <pid> GC.class_histogram snapshots an hour apart on one instance. byte[] and one application key class have grown by the same count; everything else is level. The key class has as many instances as requests served, where the team expected one per customer and report.
  3. Dump. The next nightly failure writes a dump, because -XX:+HeapDumpOnOutOfMemoryError and a dump path on a persistent volume were added after the first look.
  4. Holder. The dominator tree shows one HashMap retaining over 90% of the heap. The path to GC roots from one of its values ends at a static field of the report service.
  5. Cause. The key class has two fields and no equals or hashCode. It was a record until someone turned it into a class to add a mutable field.
  6. Fix and proof. The key becomes a record again and the cache gets a size bound. Two histograms after the deploy show 500 keys, then 500 keys.

Program outputAfter the fix: the same load with a record as the key (java -Xmx64m HeapLeak fixed)

Real JVM output
pid 4873, mode fixed
requests=  1,000  cache entries=    500  heap used=  6 MB of 64 MB
requests=  2,000  cache entries=    500  heap used=  6 MB of 64 MB
requests=  3,000  cache entries=    500  heap used=  7 MB of 64 MB
...
requests= 19,000  cache entries=    500  heap used=  7 MB of 64 MB
requests= 20,000  cache entries=    500  heap used=  7 MB of 64 MB
finished without running out of memory
500 customers, 500 entries, and heap use level at 7 MB after 20,000 requests. The leaking version died at 6,000. Verification is this: the number that grew now stays flat under the same load.

Java and Spring Boot

What to have in place before a leak happens: the JVM flags that capture evidence at the moment of failure, a script for the two-histogram comparison, and the discipline for the leak that hides best, a ThreadLocal on a pooled thread. The flags and the script's commands were run on OpenJDK 21.0.13 here. TenantContext was compiled with JDK 21.

deploy/jvm.envShell
JAVA_TOOL_OPTIONS="-Xms2g -Xmx2g \  -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/dumps \  -XX:+ExitOnOutOfMemoryError \  -Xlog:gc*:file=/var/log/reports/gc.log:time,uptime,level,tags:filecount=5,filesize=20m"

Why it is written this way

  • HeapDumpPath may be a directory; the file is then named java_pid<pid>.hprof inside it. /var/dumps must be a volume that survives the container and has at least as much free space as the heap.
  • -XX:+ExitOnOutOfMemoryError makes the JVM exit after the first OutOfMemoryError. A process that has failed an allocation is in an unknown state: some thread lost an operation halfway. In the run above the dump was written first and the exit code was 3.
  • The GC log gives the trend of heap after collection without any other tooling, and shows the full collections that free nothing just before the error.

Wrong, then right

The report cache from the lab, as it would look in a Spring Boot service. Rendering a monthly statement takes about 200 ms, 500 customers ask for one repeatedly, and the result is about 10 kB. A map from (customer, report type) to the rendered bytes is the obvious optimisation.

Incorrect implementation

src/main/java/com/example/reports/ReportService.javaJava
package com.example.reports;import java.util.Map;import java.util.concurrent.ConcurrentHashMap;import org.springframework.stereotype.Service;@Servicepublic class ReportService {    /** Looks like a value object. It is not: without equals() and hashCode(), identity is the only equality. */    static final class ReportKey {        final long customerId;        final String reportType;        ReportKey(long customerId, String reportType) {            this.customerId = customerId;            this.reportType = reportType;        }    }    private final Map<ReportKey, byte[]> cache = new ConcurrentHashMap<>();    private final ReportRenderer renderer;    public ReportService(ReportRenderer renderer) {        this.renderer = renderer;    }    public byte[] monthlyStatement(long customerId) {        ReportKey key = new ReportKey(customerId, "monthly");   // a new object on every call        return cache.computeIfAbsent(key, k -> renderer.render(k.customerId, k.reportType));    }}

Why it is written this way

  • ReportKey inherits equals and hashCode from Object: two keys are equal only if they are the same object. new ReportKey(7, "monthly") is never equal to an earlier new ReportKey(7, "monthly").
  • The service is a singleton bean, so cache lives as long as the application context: effectively a static field.
  • Nothing ever removes an entry.

Why it fails

  • Every call builds a new key that equals no existing key, so computeIfAbsent never finds an entry. The cache has a hit rate of zero: every request pays the 200 ms of rendering it was meant to save.
  • Every miss inserts a new entry of about 10 kB that is reachable from the singleton bean and therefore never collected. Heap use after collection grows by 10 MB per thousand requests. In the real run with a 64 MB heap the process died at 6,000 requests.
  • On the way there the collector works harder for less: young collections that free nothing, then repeated full collections, so latency degrades before the failure. The final OutOfMemoryError is thrown in whichever thread allocates next, often in unrelated code, which is why the stack trace of the error rarely points at the leak.
  • Tests do not catch it. A unit test that asks for one report and gets the right bytes passes. The only visible signs are a cache hit metric at zero, if anyone publishes one, and the heap trend.

Corrected implementation

src/main/java/com/example/reports/ReportService.javaJava
package com.example.reports;import java.util.Collections;import java.util.LinkedHashMap;import java.util.Map;import org.springframework.stereotype.Service;@Servicepublic class ReportService {    /** A record gets equals() and hashCode() from its components: equal contents, equal keys. */    record ReportKey(long customerId, String reportType) {}    private static final int MAX_ENTRIES = 2_000;   // 2,000 x 10 kB = 20 MB at most    private final Map<ReportKey, byte[]> cache = Collections.synchronizedMap(            new LinkedHashMap<>(256, 0.75f, true) {                @Override                protected boolean removeEldestEntry(Map.Entry<ReportKey, byte[]> eldest) {                    return size() > MAX_ENTRIES;                }            });    private final ReportRenderer renderer;    public ReportService(ReportRenderer renderer) {        this.renderer = renderer;    }    public byte[] monthlyStatement(long customerId) {        ReportKey key = new ReportKey(customerId, "monthly");        byte[] cached = cache.get(key);        if (cached != null) return cached;        byte[] rendered = renderer.render(key.customerId(), key.reportType());   // outside the map's lock        cache.put(key, rendered);        return rendered;    }    public int cachedReports() {        return cache.size();    }}

Why it is written this way

  • The record is the fix for the leak: a second request for customer 7 finds the first one's entry. In the real run the cache settled at 500 entries and 7 MB.
  • The bound is the fix for the next leak. Correct keys still grow without limit if the key space does, for example when a date range becomes part of the key. An access-ordered LinkedHashMap with removeEldestEntry evicts the least recently used entry.
  • Rendering happens outside the map's lock, so a slow render does not block other lookups. Two simultaneous misses for the same key may both render; the second put overwrites the first with identical content.
  • cachedReports() is there to be published as a gauge. An entry count that keeps rising is visible weeks before an OutOfMemoryError.
  • ReportRenderer is assumed to exist. Both variants were compiled on JDK 21 against a stand-in for the Spring annotation; the behaviour quoted comes from the lab program HeapLeak.java, which uses the same two key types.

Trade-offs

  • A bounded cache evicts. With more than 2,000 active keys some requests render again, so the bound has to come from the memory you can spare and the hit rate you need, and both should be measured.
  • Cached statements go stale. The leaking version never served a stale report because it never served a cached one. A working cache needs an expiry or an invalidation when the underlying data changes.
  • Collections.synchronizedMap puts one lock around every lookup. At high request rates that lock becomes the bottleneck; a cache library with size-based eviction avoids it at the price of a dependency.
  • A record's components are final. If the key needs to change after creation, it should not be a map key at all: a key whose hash changes while it is in a map is lost in it, which is the same leak by another route.

Configuration

The flags and commands used to capture evidence. Defaults are those of OpenJDK 21 as documented in the java and jcmd command references and printed by java -XX:+PrintFlagsFinal -version on 21.0.13.

-XX:+HeapDumpOnOutOfMemoryErrorDefault Disabled
What it does
Write an HPROF heap dump when a java.lang.OutOfMemoryError is thrown.
When to change it
Enable it on every service. It costs nothing until it fires, and the dump at the moment of failure cannot be obtained afterwards.
-XX:HeapDumpPathDefault `java_pid<pid>.hprof` in the working directory
What it does
File or directory for the dump.
When to change it
Point it at a persistent volume with free space of at least the heap size. In a container the working directory usually disappears with the failed container.
-XX:+ExitOnOutOfMemoryErrorDefault Disabled
What it does
Exit the JVM when an OutOfMemoryError is first thrown.
When to change it
Enable it for services managed by an orchestrator. A process that survives an OutOfMemoryError is unreliable; a restart is cleaner. The heap dump is still written first.
-XX:MaxMetaspaceSizeDefault Unlimited
What it does
Upper limit on native memory for class metadata.
When to change it
Setting a generous limit turns a class loader leak into a clear OutOfMemoryError: Metaspace instead of slow growth of the process until the container is killed.
-XX:MaxDirectMemorySizeDefault 0, meaning the JVM chooses the limit itself
What it does
Upper limit on memory reserved by direct ByteBuffers.
When to change it
Set it explicitly for services that use direct buffers heavily (network frameworks), so that the container limit accounts for it.
jcmd <pid> GC.class_histogramDefault Live objects only (`-all` is false)
What it does
Instance count and bytes per class.
When to change it
Take two, some time apart, and compare. It forces a full collection and pauses the application: one instance, not in a loop.
jcmd <pid> GC.heap_dump <file>Default Live objects only (`-all` is false), uncompressed, does not overwrite an existing file
What it does
Write an HPROF dump of a running process.
When to change it
Use it when you cannot wait for the failure. -gz=1 compresses the file. Expect a pause that grows with the heap, and handle the file as sensitive data.

Hands-on exercise

From a rising heap to the reference that holds it

Watch a heap that grows between collections, take two histograms to find which classes grow, follow the path to the GC root to find what holds them, then apply the fix and confirm that the growth has stopped.

Steps · 0 of 7 done

A report cache in a 64 MB heap

Simulation · not a real JVM

Let the requests run, take two histograms some time apart and compare them, then follow the path from a leaked object to its GC root. Switch the key type to see the fix.

Take it further in the simulation: Heap leak: the cache that never hits

What can go wrong

Answering every OutOfMemoryError with more heap
Symptom
The interval between restarts doubles each time the heap is doubled, and the restarts never stop.
Cause
The live set grows with time, not with load. Any heap size is reached eventually.
Fix
Plot heap after collection against time at constant traffic. If it rises, find the growing class with two histograms before touching -Xmx.
Taking a heap dump for an error that is not about the heap
Symptom
The dump shows a heap that is a quarter full. The error was Metaspace or a direct buffer message.
Cause
The message names the area that ran out. Class metadata and direct buffers are native memory and barely appear in a heap dump.
Fix
Read the text after OutOfMemoryError: first. For Metaspace use jcmd <pid> VM.classloader_stats; for native memory use native memory tracking.
No dump when the failure finally happens
Symptom
The service died overnight with Java heap space and there is nothing to analyse except a stack trace in unrelated code.
Cause
-XX:+HeapDumpOnOutOfMemoryError was not set, or the dump was written to a container file system that was deleted with the container.
Fix
Set the flag and a -XX:HeapDumpPath on a persistent volume as part of the standard JVM options, and check that the volume has room for a file the size of the heap.
Reading the histogram top to bottom
Symptom
An investigation concludes that "the problem is byte arrays and Strings" and stalls there.
Cause
byte[], String and Object[] lead every histogram. A single snapshot shows what is large, not what is growing or who holds it.
Fix
Compare two snapshots and look for the application class that grows by the same count as the generic ones. Then use a dump or JFR old object samples to find the holder.
ThreadLocal values left on pooled threads
Symptom
Memory grows in steps as the request pool reaches its maximum size, then stays high; occasionally a request sees another request's context.
Cause
A value was set on a pool thread and not removed. The thread, kept alive by the pool, holds it indefinitely.
Fix
Always pair set with remove() in a finally block, or hide the pair behind one method as TenantContext.callAs does.

How to troubleshoot

Heap after collection rises steadily at constant traffic

Check

jcmd <pid> GC.class_histogram | head -15   (twice, some minutes apart)
How to read it
Classes whose instance count rose between the two snapshots are the leak. Look for an application class growing by the same number as [B or java.util.HashMap$Node.
java.lang.OutOfMemoryError: Java heap space in the log

Check

grep -E 'Pause Full|OutOfMemoryError' gc.log | tail -20   and   ls -la /var/dumps/*.hprof
How to read it
Full collections that free almost nothing just before the error (63M->63M) mean the heap was full of reachable objects. The .hprof file is the dump to open in a heap analyser; its size is about the heap in use.
You need to know what holds the leaked objects without a full dump

Check

jcmd <pid> JFR.dump path-to-gc-roots=true filename=/tmp/leak.jfr   then   jfr print --events jdk.OldObjectSample /tmp/leak.jfr
How to read it
Needs a flight recording that is already running. Each event shows a sampled long-lived object, the reference chain to its GC root in the object block, and with the profile settings the stack trace of its allocation.
java.lang.OutOfMemoryError: Metaspace

Check

jcmd <pid> VM.classloader_stats
How to read it
One row per class loader with the number of classes it loaded. Hundreds of loaders of the same type, where you expect one, mean loaders are created and kept alive: a class loader leak.
A Spring Boot service where you have HTTP access but no shell

Check

GET /actuator/metrics/jvm.gc.live.data.size   and, authenticated, GET /actuator/heapdump
How to read it
The first is the size of the old generation after the last collection: its trend is the leak test. The second returns an HPROF dump on HotSpot. It stops the application while it runs and exposes everything in memory, so it must be secured.

Key takeaways

  • A leak on a managed heap is a set of objects that is reachable, useless and growing. The collector is working correctly; a reference is wrong.
  • The test is the trend of heap after collection at constant load. Flat and high is a sizing question; rising is a leak, and more heap only postpones it.
  • Read the message: Java heap space is the heap, Metaspace and direct buffer errors are not, and a heap dump helps only with the first.
  • Two class histograms some time apart show what grows. The application class growing in step with byte[] or HashMap$Node names the leak.
  • Set -XX:+HeapDumpOnOutOfMemoryError with a persistent -XX:HeapDumpPath before you need it, and treat dumps as sensitive data.
  • In a heap analyser, the dominator tree shows what retains the memory and the path to GC roots shows why it cannot be collected. JFR old object samples give the same chain from a sample.
  • Map keys need equals and hashCode (use records), caches need a bound, and ThreadLocal values need remove() in finally.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 4

A service with -Xmx4g is restarted by its orchestrator about every 30 hours with OutOfMemoryError: Java heap space. After the heap was raised to 8 GB the restarts come about every 60 hours. Traffic has the same daily shape every day. Heap after full collection is 1.1 GB an hour after start and grows by about 130 MB per hour.

What is the right next step?

Scenario 2 of 4

Two histograms taken 10 minutes apart show: [B from 410,000 to 530,000 instances, com.shop.PriceKey from 400,000 to 520,000, java.util.concurrent.ConcurrentHashMap$Node from 405,000 to 525,000. The catalogue has 30,000 products and each price lookup uses new PriceKey(productId, currency) against a cache.

What is the most likely defect?

Scenario 3 of 4

A service fails with java.lang.OutOfMemoryError: Metaspace. -XX:MaxMetaspaceSize=256m is set. A heap dump taken on the error shows the heap 30% full. The service loads tenant-specific plugins and reloads them when a tenant changes its configuration, a few hundred times a day.

Where should the investigation go?

Scenario 4 of 4

A team needs a heap dump from a production instance with a 24 GB heap that is leaking slowly. The instance serves live traffic, the node has 20 GB of free disk, and the service handles card payments.

Which plan is sound?

References

References: Java SE 21 Troubleshooting Guide: Troubleshoot Memory Leaks (OutOfMemoryError messages, histograms, heap dumps, JFR) ↗ · The jcmd command, JDK 21 (GC.class_histogram, GC.heap_dump, GC.heap_info, JFR.dump) ↗ · The java command, JDK 21 (-XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath, -XX:MaxMetaspaceSize) ↗ · ThreadLocal (Java SE 21 API) ↗ · Eclipse Memory Analyzer help: Basic Tutorial (histogram, dominator tree, path to GC roots, leak suspects) ↗ · Eclipse Memory Analyzer help: Shallow vs. Retained Heap ↗ · Spring Boot 3.5 Actuator API: Heap Dump (heapdump) ↗ · Report a problem