Heap leak: the cache that never hits
Send requests at a cache whose key has no equals() and hashCode(). Watch the heap floor rise to OutOfMemoryError, then find the cause with two histograms and a heap dump.
The situation
A Spring Boot 3 billing service renders monthly statements, about 10 kB each, and caches them in a map inside a singleton bean. Only 500 customers use the feature, so the cache should stay near 5 MB. The pod runs with -Xmx64m in the lab (think -Xmx2g in production: the shape is the same, only slower). After each deployment memory climbs steadily, latency degrades over the last hour, and the pod is killed and restarted with java.lang.OutOfMemoryError: Java heap space. With real traffic that happens about once a day, usually at night. Restarting fixes it, every time, for a day.
Read first: Memory leaks and heap dumps, Heap, stacks, Metaspace and native memory
Break it in the simulation
A report cache on a 64 MB heap
Simulation · not a real JVMSend requests in batches of 1,000 and compare the cache entries with the number of customers. Then investigate as you would on a live process: two histograms, then the heap dump.
Guided run · 0 of 6 stages done
- Under Cache key choose ReportKey record.
- Press Send 1,000 requests three or four times.
Look for
- Cache entries stops at 500, the same as Different customers.
- The chart has teeth, but the thick line (heap used right after a collection) is flat at about 6 MB. Heap after a collection does not move.
- The Verdict reads Stable.
- Under Cache key choose LeakyKey class. The JVM restarts with an empty cache.
- Press Send 1,000 requests three times.
Look for
- Cache entries equals Requests served: 3,000 entries for 500 customers. Every lookup missed.
- The teeth are still there, but each collection ends higher than the last. The floor rises about 10 MB per 1,000 requests.
- Nothing has failed yet. In production this is the quiet part of the day.
- Press Take class histogram now, at 3,000 requests.
- Press Send 1,000 requests three more times, to 6,000.
- Press Take class histogram again and read the two growth columns.
Look for
- Three rows grew by the same count:
[B(byte arrays) +3,000 instances and about +30 MB,java.util.HashMap$Node+3,000, andHeapLeak$LeakyKey+3,000. The map's bucket array[Ljava.util.HashMap$Node;grew in bytes only: it doubled. java.lang.Stringandjava.lang.Classdid not grow at all. They are large, but they are not the leak.- An application class with 6,000 instances when there are 500 customers is the lead:
LeakyKeyis a key, and keys are supposed to be unique.
- Press Send 1,000 requests once more.
- Press Open heap dump.
Look for
- The run stops at about 6,250 requests with
java.lang.OutOfMemoryError: Java heap space, and the send button is disabled: the process is gone. - In the dominator tree one
java.util.HashMapretains about 98% of the heap. Its shallow size is 48 bytes. - The path to the GC root reads:
class HeapLeak→static CACHE→HashMap→HashMap$Node[]→HashMap$Node→ key andbyte[10240]. - The last box states the cause: thousands of key objects, 500 different values among them.
- Press Reset, then choose ReportKey record under Cache key.
- Press Send 1,000 requests seven or more times: past the point where the other version died.
Look for
- Cache entries stays at 500 and the floor is flat.
- No OutOfMemoryError, however many batches you send.
- With the record key, press Take class histogram, send 2,000 more requests, and take another.
- Then switch back to LeakyKey class, move Heap size (-Xmx) to 128 MB and send requests until it fails.
Look for
- With the record: every growth column is 0.
HeapLeak$ReportKeyhas 500 instances in both snapshots. - With the leaking key and twice the heap: the same failure at about 12,750 requests instead of 6,250. A bigger heap buys time and nothing else.
Wrong, then right
Incorrect implementation
package com.example.billing;import java.time.YearMonth;import java.util.Map;import java.util.concurrent.ConcurrentHashMap;import org.springframework.stereotype.Service;@Servicepublic class StatementService { /** Looks like a value. It is not: equals() and hashCode() are the ones from Object. */ static final class StatementKey { final long customerId; final YearMonth month; StatementKey(long customerId, YearMonth month) { this.customerId = customerId; this.month = month; } } // A singleton bean lives as long as the application, and so does everything in this map. private final Map<StatementKey, byte[]> cache = new ConcurrentHashMap<>(); private final StatementRenderer renderer; public StatementService(StatementRenderer renderer) { this.renderer = renderer; } public byte[] statement(long customerId, YearMonth month) { // A new key object per request: never equal to a stored one, so this always misses and always adds. StatementKey key = new StatementKey(customerId, month); return cache.computeIfAbsent(key, k -> renderer.render(k.customerId, k.month)); }}interface StatementRenderer { byte[] render(long customerId, YearMonth month);}Why it is written this way
Object.equals()compares identity andObject.hashCode()is not derived from the fields. TwoStatementKeyobjects with the same customer and month are different keys.- The code is functionally correct: every caller gets the right statement. No test that checks results will catch it. Only the memory graph does.
- The map has no size limit and no expiry. Even with a correct key it would grow with every new customer and every new month.
Why it fails
computeIfAbsentlooks the key up withhashCode()andequals(). A freshly builtStatementKeyhas its own identity hash and equals only itself, so the lookup misses, the statement is rendered again, and a new entry is added. The cache never returns a hit, so it also saves no work.- The map is referenced from a singleton bean, which the application context holds for the life of the process. Every entry is therefore reachable, and a collector frees only what is unreachable. Each collection removes the request garbage and leaves all entries: the heap after collection rises by one entry per request.
- Near the limit the JVM collects more and more often and frees less and less, which is why latency degrades before the crash. When a full collection cannot make room for the next allocation, the thread that asked gets
java.lang.OutOfMemoryError: Java heap space. In the real run that was after 6,000 entries of 10 kB in a 64 MB heap. - The error is thrown in whichever thread allocates next, often one that has nothing to do with the cache. The stack trace of the OutOfMemoryError therefore points at a victim, not at the cause. The heap dump points at the cause.
Corrected implementation
package com.example.billing;import java.time.YearMonth;import java.util.LinkedHashMap;import java.util.Map;import org.springframework.stereotype.Service;@Servicepublic class StatementService { /** A record gets equals() and hashCode() from its components. */ record StatementKey(long customerId, YearMonth month) {} private static final int MAX_ENTRIES = 2_000; // 2,000 x 10 kB = 20 MB at most // Access-ordered LinkedHashMap: the least recently used entry is dropped when the limit is passed. private final Map<StatementKey, byte[]> cache = new LinkedHashMap<>(256, 0.75f, true) { @Override protected boolean removeEldestEntry(Map.Entry<StatementKey, byte[]> eldest) { return size() > MAX_ENTRIES; } }; private final StatementRenderer renderer; public StatementService(StatementRenderer renderer) { this.renderer = renderer; } public byte[] statement(long customerId, YearMonth month) { StatementKey key = new StatementKey(customerId, month); synchronized (cache) { byte[] cached = cache.get(key); if (cached != null) { return cached; } } byte[] rendered = renderer.render(customerId, month); // outside the lock: rendering is slow synchronized (cache) { cache.putIfAbsent(key, rendered); return cache.get(key); } }}// Started with: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/dumps -XX:+ExitOnOutOfMemoryErrorinterface StatementRenderer { byte[] render(long customerId, YearMonth month);}Why it is written this way
- The record is the fix for this bug: two keys with the same customer and month are now equal and hash alike, so the second request hits.
- The size limit is the fix for the next bug. A cache without a bound is a leak that has not happened yet. 2,000 entries of 10 kB is a known 20 MB.
- An access-ordered
LinkedHashMapchanges its internal order onget, so every access needs the lock. Rendering happens outside it, which means two threads may render the same statement once;putIfAbsentkeeps the first. -XX:+HeapDumpOnOutOfMemoryErroris off by default. Without it the evidence is gone when the pod restarts.-XX:HeapDumpPathmust point at storage that survives the restart.- Both classes compile on JDK 21 with a stand-in for the
@Serviceannotation. They were not run inside a Spring Boot application.
Trade-offs
- A bounded cache evicts. A statement that was dropped is rendered again on the next request, so the limit trades memory for CPU and latency on a miss. Size it from the hit rate you measure.
- One lock around the map serialises every lookup. That is fine at a few hundred requests per second with a fast
get; at high rates use a concurrent cache library with size-based eviction instead. - A heap dump is as large as the live heap and the JVM is stopped while it is written. On a multi-gigabyte heap that is seconds to minutes, and the file contains customer data: treat it like a database backup.
-XX:+ExitOnOutOfMemoryErrorends the process at the first OutOfMemoryError so the orchestrator restarts it cleanly. The alternative is a JVM that limps on with some threads dead, which is harder to notice and to reason about.
Run it on a real JVM
HeapLeak.java keeps a static HashMap of 10 kB byte arrays and serves requests for 500 customers. In leak mode the key is a class without equals() and hashCode(), and the program sleeps 1.5 s after every 1,000 requests so you have time to look at it with jcmd. In fixed mode the key is a record. Run both under -Xmx64m.
With Docker (CPU and memory limits)
cd labs/jvm/heap-leakdocker build -t jvm-lab-heap-leak .mkdir -p dumps# broken: runs for about 10 seconds, then OutOfMemoryError and a heap dump in ./dumpsdocker run --rm --name heap-leak --cpus=2 --memory=512m -v "$PWD/dumps:/dumps" jvm-lab-heap-leak -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps/leak.hprof HeapLeak leak# while it runs, from a second terminal: two histograms a few seconds apart (java is pid 1 in the container)docker exec heap-leak jcmd 1 GC.class_histogram | head -12docker exec heap-leak jcmd 1 GC.class_histogram | head -12# optional: a dump on demand instead of waiting for the errordocker exec heap-leak jcmd 1 GC.heap_dump /dumps/manual.hprof# fixed: the same requests with the record keydocker run --rm --cpus=2 --memory=512m jvm-lab-heap-leak -Xmx64m HeapLeak fixedWithout Docker, on a local JDK 21
cd labs/jvm/heap-leakjavac -d out HeapLeak.javajava -XX:ActiveProcessorCount=2 -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/leak.hprof -cp out HeapLeak leak# second terminal; the program prints its pid on the first linejcmd <pid> GC.class_histogram | head -12jcmd <pid> GC.class_histogram | head -12java -XX:ActiveProcessorCount=2 -Xmx64m -cp out HeapLeak fixedThe program: HeapLeak.java
import java.util.HashMap;import java.util.Map;/** * Lab 5: heap leak investigation. * * A "cache" keyed by a class that does not define equals() and hashCode(). Every lookup misses, * every miss adds an entry, and nothing is ever removed. * * javac -d out HeapLeak.java * java -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/leak.hprof -cp out HeapLeak leak * java -Xmx64m -cp out HeapLeak fixed * * While it runs: jcmd <pid> GC.class_histogram | head -12 */public class HeapLeak { /** The bug: two keys for the same customer and report are different objects, so they are never equal. */ static final class LeakyKey { final long customerId; final String report; LeakyKey(long customerId, String report) { this.customerId = customerId; this.report = report; } } /** The fix: a record gets equals() and hashCode() from its components. */ record ReportKey(long customerId, String report) {} static final Map<Object, byte[]> CACHE = new HashMap<>(); static byte[] render(Object key) { return CACHE.computeIfAbsent(key, k -> new byte[10 * 1024]); // a 10 kB rendered report } public static void main(String[] args) throws Exception { boolean leak = args.length == 0 || args[0].equals("leak"); System.out.println("pid " + ProcessHandle.current().pid() + ", mode " + (leak ? "leak" : "fixed")); Runtime rt = Runtime.getRuntime(); for (long request = 1; request <= 400_000; request++) { long customer = request % 500; // only 500 different customers ever ask render(leak ? new LeakyKey(customer, "monthly") : new ReportKey(customer, "monthly")); if (request % 1_000 == 0) { System.out.printf("requests=%,7d cache entries=%,7d heap used=%3d MB of %d MB%n", request, CACHE.size(), (rt.totalMemory() - rt.freeMemory()) / (1024 * 1024), rt.maxMemory() / (1024 * 1024)); Thread.sleep(leak ? 1_500 : 0); // slow enough to look at the process before it dies } if (!leak && request == 20_000) break; } System.out.println("finished without running out of memory"); }}The Dockerfile
# Java 21 lab image. Build: docker build -t jvm-lab-heap-leak .FROM eclipse-temurin:21-jdkWORKDIR /labCOPY HeapLeak.java .RUN javac -d out HeapLeak.java# The JVM reads the container's CPU and memory limits (container support is on by default),# so --cpus and --memory on "docker run" change what Runtime.availableProcessors() and the default heap size report.ENTRYPOINT ["java", "-cp", "out"]CMD ["HeapLeak"]$ java -XX:ActiveProcessorCount=2 -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/leak.hprof -cp out HeapLeak leak
Program outputThe leaking run
Real JVM outputpid 4790, mode leak requests= 1,000 cache entries= 1,000 heap used= 11 MB of 64 MB requests= 2,000 cache entries= 2,000 heap used= 21 MB of 64 MB requests= 3,000 cache entries= 3,000 heap used= 31 MB of 64 MB requests= 4,000 cache entries= 4,000 heap used= 41 MB of 64 MB requests= 5,000 cache entries= 5,000 heap used= 50 MB of 64 MB requests= 6,000 cache entries= 6,000 heap used= 60 MB of 64 MB java.lang.OutOfMemoryError: Java heap space Dumping heap to /tmp/leak.hprof ... Heap dump file created [69295888 bytes in 0.024 secs] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"
$ jcmd 4790 GC.class_histogram | head -12
HeapClass histogram at 3,000 requests
Real JVM output4790: num #instances #bytes class name (module) ------------------------------------------------------- 1: 12440 31168384 [B (java.base@21.0.13) 2: 9357 224568 java.lang.String (java.base@21.0.13) 3: 882 208448 [Ljdk.internal.vm.FillerElement; (java.base@21.0.13) 4: 1562 192880 java.lang.Class (java.base@21.0.13) 5: 5417 173344 java.util.HashMap$Node (java.base@21.0.13) 6: 1561 161456 [Ljava.lang.Object; (java.base@21.0.13) 7: 3000 72000 HeapLeak$LeakyKey 8: 217 56856 [C (java.base@21.0.13) 9: 275 51792 [Ljava.util.HashMap$Node; (java.base@21.0.13)
[B is byte[]: 12,440 arrays, 31 MB, far ahead of everything else. Further down, 3,000 HeapLeak$LeakyKey. One snapshot says what is big. It cannot say what is growing.$ jcmd 4790 GC.class_histogram | head -12
HeapClass histogram at 6,000 requests
Real JVM output4790: num #instances #bytes class name (module) ------------------------------------------------------- 1: 15424 61935760 [B (java.base@21.0.13) 2: 1006 411448 [Ljdk.internal.vm.FillerElement; (java.base@21.0.13) 3: 7287 233184 java.util.HashMap$Node (java.base@21.0.13) 4: 9341 224184 java.lang.String (java.base@21.0.13) 5: 1553 191808 java.lang.Class (java.base@21.0.13) 6: 1507 159928 [Ljava.lang.Object; (java.base@21.0.13) 7: 6000 144000 HeapLeak$LeakyKey 8: 273 59952 [Ljava.util.HashMap$Node; (java.base@21.0.13) 9: 217 56856 [C (java.base@21.0.13)
[B went from 31.2 MB to 61.9 MB, HeapLeak$LeakyKey from 3,000 to 6,000 instances, HashMap$Node grew too. String and Class did not move. Byte arrays, map nodes and one application class growing in step means a map of byte arrays keyed by that class.$ java -XX:ActiveProcessorCount=2 -Xmx64m -cp out HeapLeak fixed
Program outputThe fixed run
Real JVM outputpid 4873, mode fixed requests= 1,000 cache entries= 500 heap used= 6 MB of 64 MB requests= 2,000 cache entries= 500 heap used= 6 MB of 64 MB requests= 3,000 cache entries= 500 heap used= 7 MB of 64 MB requests= 4,000 cache entries= 500 heap used= 7 MB of 64 MB requests= 5,000 cache entries= 500 heap used= 7 MB of 64 MB requests= 6,000 cache entries= 500 heap used= 7 MB of 64 MB requests= 7,000 cache entries= 500 heap used= 7 MB of 64 MB
Real run and simulation: what each one shows
- The captured files come from running the program directly on OpenJDK 21.0.13 with
-XX:ActiveProcessorCount=2, not in Docker. The Docker commands above were not executed when this was written. - The real program died somewhere between 6,000 and 7,000 entries, after several full collections that freed nothing. The simulation stops at a fixed point, about 6,250 entries, without that slow final phase.
- A real histogram has hundreds of rows, including JVM-internal ones such as
jdk.internal.vm.FillerElement[](space G1 has filled, not application data). The simulation shows the eight rows that matter and keeps the JDK's own classes constant. - The simulated histogram matches the real one closely for the leak itself:
[B31,168,384 bytes at 3,000 entries in both. Its count ofHashMap$Nodeat 6,000 entries is higher than the real one, because the real JVM had dropped some of its own maps by then. - The real run produced a 69 MB
.hproffile that you open in a heap analyser to get the dominator tree. The lab does not include that file or a screenshot of it; the widget's heap dump view is an illustration of what the tool shows. - Taking a histogram on a real JVM runs a full collection first (
Pause Full (Heap Inspection Initiated GC)in the GC log) and pauses the application for that long. The simulation shows the drop in the chart but not the pause.
How you meet this in production
- The first symptom is a restart pattern: the service is killed or exits with
java.lang.OutOfMemoryError: Java heap spaceat an interval that follows traffic, and a restart always helps. Hours before that, latency rises as collections get more frequent. - The graph that shows a leak is heap used after collection over days, not heap used. Heap used is a sawtooth on any healthy JVM. In a GC log, read the after-value of the old-generation or full collections; if each is higher than the last, something is accumulating.
- On the live process take two histograms some time apart with
jcmd <pid> GC.class_histogramand compare them. Each one runs a full collection, so expect a pause. - Start every service with
-XX:+HeapDumpOnOutOfMemoryErrorand a-XX:HeapDumpPathon storage that outlives the pod. Open the dump in a heap analyser, sort by retained size, and follow the path to the GC root from the largest object. - JFR can also point at the leak without a dump: its
jdk.OldObjectSampleevents record sampled long-lived objects, with the reference chain when the recording is dumped withpath-to-gc-roots=true. - Practise the diagnosis on Restarts every night.
References
References: Java SE 21 Troubleshooting Guide: Troubleshoot Memory Leaks ↗ · Java SE 21 Tool Specifications: the jcmd command (GC.class_histogram, GC.heap_dump) ↗ · Java SE 21 Tool Specifications: the java command (-XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath) ↗ · Report a problem