Skip to content
Intermediate25 minSimulation 5 · Java 21

Heap leak: the cache that never hits

Send requests at a cache whose key has no equals() and hashCode(). Watch the heap floor rise to OutOfMemoryError, then find the cause with two histograms and a heap dump.

The situation

A Spring Boot 3 billing service renders monthly statements, about 10 kB each, and caches them in a map inside a singleton bean. Only 500 customers use the feature, so the cache should stay near 5 MB. The pod runs with -Xmx64m in the lab (think -Xmx2g in production: the shape is the same, only slower). After each deployment memory climbs steadily, latency degrades over the last hour, and the pod is killed and restarted with java.lang.OutOfMemoryError: Java heap space. With real traffic that happens about once a day, usually at night. Restarting fixes it, every time, for a day.

Read first: Memory leaks and heap dumps, Heap, stacks, Metaspace and native memory

Break it in the simulation

A report cache on a 64 MB heap

Simulation · not a real JVM

Send requests in batches of 1,000 and compare the cache entries with the number of customers. Then investigate as you would on a live process: two histograms, then the heap dump.

Guided run · 0 of 6 stages done

    1. Under Cache key choose ReportKey record.
    2. Press Send 1,000 requests three or four times.

    Look for

    • Cache entries stops at 500, the same as Different customers.
    • The chart has teeth, but the thick line (heap used right after a collection) is flat at about 6 MB. Heap after a collection does not move.
    • The Verdict reads Stable.
    1. Under Cache key choose LeakyKey class. The JVM restarts with an empty cache.
    2. Press Send 1,000 requests three times.

    Look for

    • Cache entries equals Requests served: 3,000 entries for 500 customers. Every lookup missed.
    • The teeth are still there, but each collection ends higher than the last. The floor rises about 10 MB per 1,000 requests.
    • Nothing has failed yet. In production this is the quiet part of the day.
    1. Press Take class histogram now, at 3,000 requests.
    2. Press Send 1,000 requests three more times, to 6,000.
    3. Press Take class histogram again and read the two growth columns.

    Look for

    • Three rows grew by the same count: [B (byte arrays) +3,000 instances and about +30 MB, java.util.HashMap$Node +3,000, and HeapLeak$LeakyKey +3,000. The map's bucket array [Ljava.util.HashMap$Node; grew in bytes only: it doubled.
    • java.lang.String and java.lang.Class did not grow at all. They are large, but they are not the leak.
    • An application class with 6,000 instances when there are 500 customers is the lead: LeakyKey is a key, and keys are supposed to be unique.
    1. Press Send 1,000 requests once more.
    2. Press Open heap dump.

    Look for

    • The run stops at about 6,250 requests with java.lang.OutOfMemoryError: Java heap space, and the send button is disabled: the process is gone.
    • In the dominator tree one java.util.HashMap retains about 98% of the heap. Its shallow size is 48 bytes.
    • The path to the GC root reads: class HeapLeak → static CACHE → HashMap → HashMap$Node[] → HashMap$Node → key and byte[10240].
    • The last box states the cause: thousands of key objects, 500 different values among them.
    1. Press Reset, then choose ReportKey record under Cache key.
    2. Press Send 1,000 requests seven or more times: past the point where the other version died.

    Look for

    • Cache entries stays at 500 and the floor is flat.
    • No OutOfMemoryError, however many batches you send.
    1. With the record key, press Take class histogram, send 2,000 more requests, and take another.
    2. Then switch back to LeakyKey class, move Heap size (-Xmx) to 128 MB and send requests until it fails.

    Look for

    • With the record: every growth column is 0. HeapLeak$ReportKey has 500 instances in both snapshots.
    • With the leaking key and twice the heap: the same failure at about 12,750 requests instead of 6,250. A bigger heap buys time and nothing else.

Wrong, then right

Incorrect implementation

src/main/java/com/example/billing/StatementService.javaJava
package com.example.billing;import java.time.YearMonth;import java.util.Map;import java.util.concurrent.ConcurrentHashMap;import org.springframework.stereotype.Service;@Servicepublic class StatementService {    /** Looks like a value. It is not: equals() and hashCode() are the ones from Object. */    static final class StatementKey {        final long customerId;        final YearMonth month;        StatementKey(long customerId, YearMonth month) {            this.customerId = customerId;            this.month = month;        }    }    // A singleton bean lives as long as the application, and so does everything in this map.    private final Map<StatementKey, byte[]> cache = new ConcurrentHashMap<>();    private final StatementRenderer renderer;    public StatementService(StatementRenderer renderer) {        this.renderer = renderer;    }    public byte[] statement(long customerId, YearMonth month) {        // A new key object per request: never equal to a stored one, so this always misses and always adds.        StatementKey key = new StatementKey(customerId, month);        return cache.computeIfAbsent(key, k -> renderer.render(k.customerId, k.month));    }}interface StatementRenderer {    byte[] render(long customerId, YearMonth month);}

Why it is written this way

  • Object.equals() compares identity and Object.hashCode() is not derived from the fields. Two StatementKey objects with the same customer and month are different keys.
  • The code is functionally correct: every caller gets the right statement. No test that checks results will catch it. Only the memory graph does.
  • The map has no size limit and no expiry. Even with a correct key it would grow with every new customer and every new month.

Why it fails

  • computeIfAbsent looks the key up with hashCode() and equals(). A freshly built StatementKey has its own identity hash and equals only itself, so the lookup misses, the statement is rendered again, and a new entry is added. The cache never returns a hit, so it also saves no work.
  • The map is referenced from a singleton bean, which the application context holds for the life of the process. Every entry is therefore reachable, and a collector frees only what is unreachable. Each collection removes the request garbage and leaves all entries: the heap after collection rises by one entry per request.
  • Near the limit the JVM collects more and more often and frees less and less, which is why latency degrades before the crash. When a full collection cannot make room for the next allocation, the thread that asked gets java.lang.OutOfMemoryError: Java heap space. In the real run that was after 6,000 entries of 10 kB in a 64 MB heap.
  • The error is thrown in whichever thread allocates next, often one that has nothing to do with the cache. The stack trace of the OutOfMemoryError therefore points at a victim, not at the cause. The heap dump points at the cause.

Corrected implementation

src/main/java/com/example/billing/StatementService.javaJava
package com.example.billing;import java.time.YearMonth;import java.util.LinkedHashMap;import java.util.Map;import org.springframework.stereotype.Service;@Servicepublic class StatementService {    /** A record gets equals() and hashCode() from its components. */    record StatementKey(long customerId, YearMonth month) {}    private static final int MAX_ENTRIES = 2_000;   // 2,000 x 10 kB = 20 MB at most    // Access-ordered LinkedHashMap: the least recently used entry is dropped when the limit is passed.    private final Map<StatementKey, byte[]> cache = new LinkedHashMap<>(256, 0.75f, true) {        @Override        protected boolean removeEldestEntry(Map.Entry<StatementKey, byte[]> eldest) {            return size() > MAX_ENTRIES;        }    };    private final StatementRenderer renderer;    public StatementService(StatementRenderer renderer) {        this.renderer = renderer;    }    public byte[] statement(long customerId, YearMonth month) {        StatementKey key = new StatementKey(customerId, month);        synchronized (cache) {            byte[] cached = cache.get(key);            if (cached != null) {                return cached;            }        }        byte[] rendered = renderer.render(customerId, month);   // outside the lock: rendering is slow        synchronized (cache) {            cache.putIfAbsent(key, rendered);            return cache.get(key);        }    }}// Started with: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/dumps -XX:+ExitOnOutOfMemoryErrorinterface StatementRenderer {    byte[] render(long customerId, YearMonth month);}

Why it is written this way

  • The record is the fix for this bug: two keys with the same customer and month are now equal and hash alike, so the second request hits.
  • The size limit is the fix for the next bug. A cache without a bound is a leak that has not happened yet. 2,000 entries of 10 kB is a known 20 MB.
  • An access-ordered LinkedHashMap changes its internal order on get, so every access needs the lock. Rendering happens outside it, which means two threads may render the same statement once; putIfAbsent keeps the first.
  • -XX:+HeapDumpOnOutOfMemoryError is off by default. Without it the evidence is gone when the pod restarts. -XX:HeapDumpPath must point at storage that survives the restart.
  • Both classes compile on JDK 21 with a stand-in for the @Service annotation. They were not run inside a Spring Boot application.

Trade-offs

  • A bounded cache evicts. A statement that was dropped is rendered again on the next request, so the limit trades memory for CPU and latency on a miss. Size it from the hit rate you measure.
  • One lock around the map serialises every lookup. That is fine at a few hundred requests per second with a fast get; at high rates use a concurrent cache library with size-based eviction instead.
  • A heap dump is as large as the live heap and the JVM is stopped while it is written. On a multi-gigabyte heap that is seconds to minutes, and the file contains customer data: treat it like a database backup.
  • -XX:+ExitOnOutOfMemoryError ends the process at the first OutOfMemoryError so the orchestrator restarts it cleanly. The alternative is a JVM that limps on with some threads dead, which is harder to notice and to reason about.

Run it on a real JVM

HeapLeak.java keeps a static HashMap of 10 kB byte arrays and serves requests for 500 customers. In leak mode the key is a class without equals() and hashCode(), and the program sleeps 1.5 s after every 1,000 requests so you have time to look at it with jcmd. In fixed mode the key is a record. Run both under -Xmx64m.

With Docker (CPU and memory limits)

labs/jvm/heap-leakShell
cd labs/jvm/heap-leakdocker build -t jvm-lab-heap-leak .mkdir -p dumps# broken: runs for about 10 seconds, then OutOfMemoryError and a heap dump in ./dumpsdocker run --rm --name heap-leak --cpus=2 --memory=512m -v "$PWD/dumps:/dumps" jvm-lab-heap-leak -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps/leak.hprof HeapLeak leak# while it runs, from a second terminal: two histograms a few seconds apart (java is pid 1 in the container)docker exec heap-leak jcmd 1 GC.class_histogram | head -12docker exec heap-leak jcmd 1 GC.class_histogram | head -12# optional: a dump on demand instead of waiting for the errordocker exec heap-leak jcmd 1 GC.heap_dump /dumps/manual.hprof# fixed: the same requests with the record keydocker run --rm --cpus=2 --memory=512m jvm-lab-heap-leak -Xmx64m HeapLeak fixed

Without Docker, on a local JDK 21

labs/jvm/heap-leakShell
cd labs/jvm/heap-leakjavac -d out HeapLeak.javajava -XX:ActiveProcessorCount=2 -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/leak.hprof -cp out HeapLeak leak# second terminal; the program prints its pid on the first linejcmd <pid> GC.class_histogram | head -12jcmd <pid> GC.class_histogram | head -12java -XX:ActiveProcessorCount=2 -Xmx64m -cp out HeapLeak fixed
The program: HeapLeak.java
labs/jvm/heap-leak/HeapLeak.javaJava
import java.util.HashMap;import java.util.Map;/** * Lab 5: heap leak investigation. * * A "cache" keyed by a class that does not define equals() and hashCode(). Every lookup misses, * every miss adds an entry, and nothing is ever removed. * *   javac -d out HeapLeak.java *   java -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/leak.hprof -cp out HeapLeak leak *   java -Xmx64m -cp out HeapLeak fixed * * While it runs:  jcmd <pid> GC.class_histogram | head -12 */public class HeapLeak {    /** The bug: two keys for the same customer and report are different objects, so they are never equal. */    static final class LeakyKey {        final long customerId;        final String report;        LeakyKey(long customerId, String report) { this.customerId = customerId; this.report = report; }    }    /** The fix: a record gets equals() and hashCode() from its components. */    record ReportKey(long customerId, String report) {}    static final Map<Object, byte[]> CACHE = new HashMap<>();    static byte[] render(Object key) {        return CACHE.computeIfAbsent(key, k -> new byte[10 * 1024]); // a 10 kB rendered report    }    public static void main(String[] args) throws Exception {        boolean leak = args.length == 0 || args[0].equals("leak");        System.out.println("pid " + ProcessHandle.current().pid() + ", mode " + (leak ? "leak" : "fixed"));        Runtime rt = Runtime.getRuntime();        for (long request = 1; request <= 400_000; request++) {            long customer = request % 500; // only 500 different customers ever ask            render(leak ? new LeakyKey(customer, "monthly") : new ReportKey(customer, "monthly"));            if (request % 1_000 == 0) {                System.out.printf("requests=%,7d  cache entries=%,7d  heap used=%3d MB of %d MB%n",                        request, CACHE.size(), (rt.totalMemory() - rt.freeMemory()) / (1024 * 1024), rt.maxMemory() / (1024 * 1024));                Thread.sleep(leak ? 1_500 : 0); // slow enough to look at the process before it dies            }            if (!leak && request == 20_000) break;        }        System.out.println("finished without running out of memory");    }}
The Dockerfile
labs/jvm/heap-leak/DockerfileDockerfile
# Java 21 lab image. Build:  docker build -t jvm-lab-heap-leak .FROM eclipse-temurin:21-jdkWORKDIR /labCOPY HeapLeak.java .RUN javac -d out HeapLeak.java# The JVM reads the container's CPU and memory limits (container support is on by default),# so --cpus and --memory on "docker run" change what Runtime.availableProcessors() and the default heap size report.ENTRYPOINT ["java", "-cp", "out"]CMD ["HeapLeak"]

$ java -XX:ActiveProcessorCount=2 -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/leak.hprof -cp out HeapLeak leak

Program outputThe leaking run

Real JVM output
pid 4790, mode leak
requests=  1,000  cache entries=  1,000  heap used= 11 MB of 64 MB
requests=  2,000  cache entries=  2,000  heap used= 21 MB of 64 MB
requests=  3,000  cache entries=  3,000  heap used= 31 MB of 64 MB
requests=  4,000  cache entries=  4,000  heap used= 41 MB of 64 MB
requests=  5,000  cache entries=  5,000  heap used= 50 MB of 64 MB
requests=  6,000  cache entries=  6,000  heap used= 60 MB of 64 MB
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/leak.hprof ...
Heap dump file created [69295888 bytes in 0.024 secs]

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"
Cache entries equal requests: 6,000 entries for 500 customers. Heap used rises 10 MB per 1,000 requests, to 60 of 64 MB. Then the error, and the JVM writes a 69 MB dump in 0.024 s. The last line shows the error striking again inside the handler that was reporting it: after an OutOfMemoryError nothing can be relied on.

$ jcmd 4790 GC.class_histogram | head -12

HeapClass histogram at 3,000 requests

Real JVM output
4790:
 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         12440       31168384  [B (java.base@21.0.13)
   2:          9357         224568  java.lang.String (java.base@21.0.13)
   3:           882         208448  [Ljdk.internal.vm.FillerElement; (java.base@21.0.13)
   4:          1562         192880  java.lang.Class (java.base@21.0.13)
   5:          5417         173344  java.util.HashMap$Node (java.base@21.0.13)
   6:          1561         161456  [Ljava.lang.Object; (java.base@21.0.13)
   7:          3000          72000  HeapLeak$LeakyKey
   8:           217          56856  [C (java.base@21.0.13)
   9:           275          51792  [Ljava.util.HashMap$Node; (java.base@21.0.13)
Columns: rank, instances, bytes, class. [B is byte[]: 12,440 arrays, 31 MB, far ahead of everything else. Further down, 3,000 HeapLeak$LeakyKey. One snapshot says what is big. It cannot say what is growing.

$ jcmd 4790 GC.class_histogram | head -12

HeapClass histogram at 6,000 requests

Real JVM output
4790:
 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         15424       61935760  [B (java.base@21.0.13)
   2:          1006         411448  [Ljdk.internal.vm.FillerElement; (java.base@21.0.13)
   3:          7287         233184  java.util.HashMap$Node (java.base@21.0.13)
   4:          9341         224184  java.lang.String (java.base@21.0.13)
   5:          1553         191808  java.lang.Class (java.base@21.0.13)
   6:          1507         159928  [Ljava.lang.Object; (java.base@21.0.13)
   7:          6000         144000  HeapLeak$LeakyKey
   8:           273          59952  [Ljava.util.HashMap$Node; (java.base@21.0.13)
   9:           217          56856  [C (java.base@21.0.13)
Compare with the first: [B went from 31.2 MB to 61.9 MB, HeapLeak$LeakyKey from 3,000 to 6,000 instances, HashMap$Node grew too. String and Class did not move. Byte arrays, map nodes and one application class growing in step means a map of byte arrays keyed by that class.

$ java -XX:ActiveProcessorCount=2 -Xmx64m -cp out HeapLeak fixed

Program outputThe fixed run

Real JVM output
pid 4873, mode fixed
requests=  1,000  cache entries=    500  heap used=  6 MB of 64 MB
requests=  2,000  cache entries=    500  heap used=  6 MB of 64 MB
requests=  3,000  cache entries=    500  heap used=  7 MB of 64 MB
requests=  4,000  cache entries=    500  heap used=  7 MB of 64 MB
requests=  5,000  cache entries=    500  heap used=  7 MB of 64 MB
requests=  6,000  cache entries=    500  heap used=  7 MB of 64 MB
requests=  7,000  cache entries=    500  heap used=  7 MB of 64 MB
The same requests with the record key: 500 entries from the first line to the last, heap used flat at 6 to 7 MB. The full file goes on to 20,000 requests without change.

Real run and simulation: what each one shows

  • The captured files come from running the program directly on OpenJDK 21.0.13 with -XX:ActiveProcessorCount=2, not in Docker. The Docker commands above were not executed when this was written.
  • The real program died somewhere between 6,000 and 7,000 entries, after several full collections that freed nothing. The simulation stops at a fixed point, about 6,250 entries, without that slow final phase.
  • A real histogram has hundreds of rows, including JVM-internal ones such as jdk.internal.vm.FillerElement[] (space G1 has filled, not application data). The simulation shows the eight rows that matter and keeps the JDK's own classes constant.
  • The simulated histogram matches the real one closely for the leak itself: [B 31,168,384 bytes at 3,000 entries in both. Its count of HashMap$Node at 6,000 entries is higher than the real one, because the real JVM had dropped some of its own maps by then.
  • The real run produced a 69 MB .hprof file that you open in a heap analyser to get the dominator tree. The lab does not include that file or a screenshot of it; the widget's heap dump view is an illustration of what the tool shows.
  • Taking a histogram on a real JVM runs a full collection first (Pause Full (Heap Inspection Initiated GC) in the GC log) and pauses the application for that long. The simulation shows the drop in the chart but not the pause.

How you meet this in production

  • The first symptom is a restart pattern: the service is killed or exits with java.lang.OutOfMemoryError: Java heap space at an interval that follows traffic, and a restart always helps. Hours before that, latency rises as collections get more frequent.
  • The graph that shows a leak is heap used after collection over days, not heap used. Heap used is a sawtooth on any healthy JVM. In a GC log, read the after-value of the old-generation or full collections; if each is higher than the last, something is accumulating.
  • On the live process take two histograms some time apart with jcmd <pid> GC.class_histogram and compare them. Each one runs a full collection, so expect a pause.
  • Start every service with -XX:+HeapDumpOnOutOfMemoryError and a -XX:HeapDumpPath on storage that outlives the pod. Open the dump in a heap analyser, sort by retained size, and follow the path to the GC root from the largest object.
  • JFR can also point at the leak without a dump: its jdk.OldObjectSample events record sampled long-lived objects, with the reference chain when the recording is dumped with path-to-gc-roots=true.
  • Practise the diagnosis on Restarts every night.
Investigate · PracticeThe report service restarts by itself, about once a day

References

References: Java SE 21 Troubleshooting Guide: Troubleshoot Memory Leaks ↗ · Java SE 21 Tool Specifications: the jcmd command (GC.class_histogram, GC.heap_dump) ↗ · Java SE 21 Tool Specifications: the java command (-XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath) ↗ · Report a problem