Skip to content
Intermediate25 minSimulation 4 · Java 21

GC pressure: when the live set fills the heap

Run a generational heap at 200 MB/s. Raise the live set until G1 falls back to full collections, then see what ZGC does with the same heap.

The situation

A Spring Boot 3 pricing service runs with -Xmx256m and the default collector, G1. It allocates about 200 MB per second while serving 900 requests per second. For months the p99 latency was 8 ms. Last week a feature started keeping the last 3,400 rendered quotes in memory, about 64 kB each. Nothing crashes and average latency is unchanged, but every few hundred milliseconds a batch of requests takes 20 ms longer, and the p99 is now 25 ms. CPU is no higher than before. The GC log has a line nobody had seen in this service: Pause Full (G1 Compaction Pause).

Read first: Garbage collection: G1 and ZGC, GC tuning and pause analysis, Object allocation, TLABs and escape analysis

Break it in the simulation

A 256 MB heap at 200 MB/s

Simulation · not a real JVM

It starts in the broken state: a 215 MB live set in a 256 MB heap. Press Play and watch the heap bar and the pause timeline, then change one input at a time.

Guided run · 0 of 6 stages done

    1. Move Live set to 60 MB. Leave Collector on G1, Heap size (-Xmx) at 256 MB and Allocation rate at 200 MB/s.
    2. Press Play and let about five simulated seconds pass, then press Pause.
    3. Press Step a few times and read each new line in GC log.

    Look for

    • Collections per second is about 9 and Longest pause is a few milliseconds. The first pauses of a new JVM are the longest.
    • In the Heap bar the young generation fills and empties, and the old generation's garbage is cut back after each Pause Remark and Pause Young (Mixed).
    • The Verdict says there is no full collection, and Time stopped by GC is 2 to 3%.
    1. Press Play and move Live set to 215 MB while it runs. The live set fills at about 56 MB per simulated second, so give it a few seconds.
    2. Keep watching Pauses over time.

    Look for

    • Young collections come closer together: G1 shrinks the young generation as free space runs out.
    • Tall bars appear in the timeline, labelled Full pause in the legend, and Longest pause jumps to between 15 and 20 ms.
    • The Verdict names the cause: the live set is 84% of the heap.
    1. Press Pause right after a tall bar, or use Step until a line with Pause Full is the newest one in GC log.
    2. Read the three or four lines above it.

    Look for

    • A young collection tagged (Evacuation Failure) whose before and after sizes are the same, for example 253M->253M(256M): it had nowhere to copy the survivors to.
    • Then Pause Full (G1 Compaction Pause) 253M->218M(256M): the heap drops to the live set and no further. That 218 MB is the number to remember.
    • The pause of the full collection is ten times that of the young collections next to it.
    1. Choose ZGC under Collector (the JVM restarts) and press Play. Leave Live set at 215 MB.
    2. Wait until the live set has filled, about four simulated seconds.

    Look for

    • Longest pause reads under 1 ms and the timeline has no pause bars.
    • Allocation stalls is no longer 0, and Allocation Stall (main) lines appear in GC log.
    • Raise Allocation rate to 400 MB/s: the stalls get much longer and Time stopped by GC climbs. A concurrent collector needs free space to work in.
    1. Set Allocation rate back to 200 MB/s and Collector to G1.
    2. Either move Live set down to 60 MB (keep less in memory), or move Heap size (-Xmx) up to 512 MB (pay for more memory). Try both.
    3. Press Play.

    Look for

    • No full pause in the timeline and Longest pause back to a few milliseconds.
    • With the 512 MB heap there are fewer collections per second than before: the young generation is larger, so it fills less often.
    1. With G1 and a 256 MB heap, move Live set up one notch at a time (180, 200, 215, 230, 245) and let it run a few seconds at each.
    2. Finally move Live set to 256 MB or higher.

    Look for

    • Up to about 180 MB only the collection rate rises. Between 200 and 215 MB full collections appear, and at 245 MB Time stopped by GC is around a third.
    • At 256 MB the run ends with java.lang.OutOfMemoryError: Java heap space: back-to-back full collections that free almost nothing, then a failed allocation.

Wrong, then right

Incorrect implementation

src/main/java/com/example/pricing/QuoteHistoryService.javaJava
package com.example.pricing;import java.util.ArrayDeque;import java.util.Deque;import org.springframework.stereotype.Service;@Servicepublic class QuoteHistoryService {    /** "Keep the last 3,400 quotes for the comparison screen." Nobody multiplied it out. */    private static final int MAX_QUOTES = 3_400;    private final Deque<byte[]> recent = new ArrayDeque<>(MAX_QUOTES);    private final QuoteRenderer renderer;    public QuoteHistoryService(QuoteRenderer renderer) {        this.renderer = renderer;    }    public synchronized byte[] quote(QuoteRequest request) {        byte[] rendered = renderer.render(request);   // about 64 kB per quote        recent.addLast(rendered);                      // 3,400 x 64 kB = 212 MB that stays reachable        if (recent.size() > MAX_QUOTES) {            recent.removeFirst();        }        return rendered;    }    public synchronized int kept() {        return recent.size();    }}// The pod runs with: -Xmx256m// 212 MB of 256 MB is live before the next request allocates anything.interface QuoteRenderer {    byte[] render(QuoteRequest request);}record QuoteRequest(long customerId, String product) {}

Why it is written this way

  • The limit is a count of objects. The heap is a count of bytes. 3,400 entries of 64 kB are 212 MB, which is 83% of a 256 MB heap.
  • The structure is bounded, so this is not a leak: the heap after a full collection is flat. That is exactly why it is missed. The service does not die, it just pauses.
  • A singleton bean lives as long as the application. Everything it references is live data, whether anyone reads it or not.

Why it fails

  • A collector can only free what is unreachable. With 212 MB reachable, G1 has about 40 MB for the young generation, for survivors being copied, and for old garbage that concurrent marking has not got to yet.
  • Each young collection promotes surviving quotes into the old generation and pushes the same amount of older quotes out of the window. Those become garbage in the old generation, where only marking followed by mixed collections reclaims them, and that takes several collections.
  • When a young collection finds no free space to copy survivors into, it reports (Evacuation Failure). G1 then schedules a full collection: all application threads stop while the whole heap is compacted in place. In the real run each took 15 to 21 ms, against about 1 ms for the young collections around it.
  • Because the full collection ends at the live set (218 MB in the real run), the heap is nearly full again at once. The cycle repeats about twice a second, so the tail latency of the service is now the length of a full collection.
  • Switching to ZGC does not remove the cause. ZGC does its work while the application runs, so it needs headroom to allocate into during a cycle. With too little, allocating threads wait for it: the real run logged 17 Allocation Stall lines.

Corrected implementation

src/main/java/com/example/pricing/QuoteHistoryService.javaJava
package com.example.pricing;import java.util.ArrayDeque;import java.util.Deque;import org.springframework.stereotype.Service;@Servicepublic class QuoteHistoryService {    /** The history may use a quarter of the heap, whatever -Xmx is. */    private static final double HEAP_SHARE = 0.25;    private final Deque<byte[]> recent = new ArrayDeque<>();    private final QuoteRenderer renderer;    private final long budgetBytes;    private long keptBytes;    public QuoteHistoryService(QuoteRenderer renderer) {        this.renderer = renderer;        this.budgetBytes = (long) (Runtime.getRuntime().maxMemory() * HEAP_SHARE);    }    public synchronized byte[] quote(QuoteRequest request) {        byte[] rendered = renderer.render(request);        recent.addLast(rendered);        keptBytes += rendered.length;        while (keptBytes > budgetBytes && recent.size() > 1) {            keptBytes -= recent.removeFirst().length;   // evict by bytes, not by count        }        return rendered;    }    public synchronized long keptBytes() {        return keptBytes;    }}// The pod runs with://   -Xmx256m -Xlog:gc*:file=/var/log/app/gc.log:time,uptime,level,tags:filecount=5,filesize=20m// With -Xmx256m the history stops at 64 MB (about 1,000 quotes).// If the product needs all 3,400 quotes, the other honest fix is a heap of 768 MB or more.interface QuoteRenderer {    byte[] render(QuoteRequest request);}record QuoteRequest(long customerId, String product) {}

Why it is written this way

  • The budget is in bytes and is derived from Runtime.maxMemory(), so it follows the heap when someone changes -Xmx or the container limit.
  • A quarter of the heap for one structure leaves the collector most of the heap to work in. The right share depends on what else is live: measure the heap after a collection in the GC log.
  • -Xlog:gc* to a rotating file costs little and is the first thing you will want when latency moves. Without it you are guessing.
  • Both classes compile on JDK 21 with a stand-in for the @Service annotation. They were not run inside a Spring Boot application.

Trade-offs

  • A smaller history means fewer quotes to compare against. If the feature needs 3,400, the cost moves to memory: a heap where 212 MB is a modest share, which means a larger pod.
  • A larger heap makes each young collection rarer but not free: pauses follow the amount of live data copied, and a full collection of a bigger live set takes longer if it ever happens.
  • ZGC keeps pauses under a millisecond, at the cost of some throughput and of needing more free heap than G1 to stay ahead of allocation. It is the answer to a pause-time requirement, not to a heap that is too small.
  • Keeping large data outside the Java heap (a cache server, a file, a database) removes it from the live set, and adds a network call or serialisation to every read.

Run it on a real JVM

GcPressure.java is a request loop paced at 200 MB/s. Each request allocates some garbage and one 64 kB piece of session data that it keeps in a sliding window. The first argument is the size of that window in MB: the live set. Run it with 60 and with 215 under -Xmx256m and compare the GC logs; then run 215 again with generational ZGC.

With Docker (CPU and memory limits)

labs/jvm/gc-pressureShell
cd labs/jvm/gc-pressuredocker build -t jvm-lab-gc-pressure .# comfortable: live set 60 MBdocker run --rm --cpus=2 --memory=512m jvm-lab-gc-pressure -Xmx256m -Xlog:gc GcPressure 60# pressure: live set 215 MBdocker run --rm --cpus=2 --memory=512m jvm-lab-gc-pressure -Xmx256m -Xlog:gc GcPressure 215# the same pressure with generational ZGCdocker run --rm --cpus=2 --memory=512m jvm-lab-gc-pressure -Xmx256m -XX:+UseZGC -XX:+ZGenerational -Xlog:gc GcPressure 215# a longer run you can look into from a second terminal (60 seconds)docker run --rm --name gc-lab --cpus=2 --memory=512m jvm-lab-gc-pressure -Xmx256m -Xlog:gc GcPressure 215 60docker exec gc-lab jcmd 1 GC.heap_info

Without Docker, on a local JDK 21

labs/jvm/gc-pressureShell
cd labs/jvm/gc-pressurejava -XX:ActiveProcessorCount=2 -Xmx256m -Xlog:gc GcPressure.java 60java -XX:ActiveProcessorCount=2 -Xmx256m -Xlog:gc GcPressure.java 215java -XX:ActiveProcessorCount=2 -Xmx256m -XX:+UseZGC -XX:+ZGenerational -Xlog:gc GcPressure.java 215
The program: GcPressure.java
labs/jvm/gc-pressure/GcPressure.javaJava
import java.util.ArrayDeque;import java.util.Arrays;import java.util.concurrent.ThreadLocalRandom;/** * Lab 4: garbage collection and memory pressure. * * A request loop allocates short-lived buffers and keeps a sliding window of "session" data alive. * The live set is set by the first argument (MB). The closer it is to the heap limit, the harder the * collector has to work. * *   java -Xmx256m -Xlog:gc GcPressure.java 60     -> comfortable: short young collections only *   java -Xmx256m -Xlog:gc GcPressure.java 215    -> pressure: the live set nearly fills the heap, full collections appear *   java -Xmx256m -XX:+UseZGC -XX:+ZGenerational -Xlog:gc GcPressure.java 60 * * Arguments: live set in MB, seconds to run (default 8), allocation rate in MB/s (default 200). */public class GcPressure {    public static void main(String[] args) {        int liveMb = args.length > 0 ? Integer.parseInt(args[0]) : 60;        int seconds = args.length > 1 ? Integer.parseInt(args[1]) : 8;        int rateMb = args.length > 2 ? Integer.parseInt(args[2]) : 200;        int chunk = 64 * 1024;        int liveChunks = liveMb * 1024 * 1024 / chunk;        ArrayDeque<byte[]> live = new ArrayDeque<>(liveChunks);        long[] worstMs = new long[5];        long requests = 0;        long allocated = 0;        long begin = System.nanoTime();        long end = begin + seconds * 1_000_000_000L;        while (System.nanoTime() < end) {            // pace the loop: a real service gets requests at a rate, it does not allocate as fast as the CPU allows            while (allocated > (System.nanoTime() - begin) / 1_000_000_000.0 * rateMb * 1024 * 1024) Thread.onSpinWait();            long t0 = System.nanoTime();            // one "request": some garbage that dies at once, and one piece of session data that lives on            for (int i = 0; i < 8; i++) {                byte[] garbage = new byte[ThreadLocalRandom.current().nextInt(8_000, 32_000)];                garbage[0] = 1;                allocated += garbage.length;            }            byte[] session = new byte[chunk];            allocated += chunk;            live.addLast(session);            if (live.size() > liveChunks) live.removeFirst();            long ms = (System.nanoTime() - t0) / 1_000_000;            if (ms > worstMs[0]) { worstMs[0] = ms; Arrays.sort(worstMs); }            requests++;        }        Runtime rt = Runtime.getRuntime();        System.out.printf("%nlive set target %d MB, max heap %d MB%n", liveMb, rt.maxMemory() / (1024 * 1024));        System.out.printf("requests %,d in %d s, allocated %,d MB (%,d MB/s)%n", requests, seconds, allocated / (1024 * 1024), allocated / (1024 * 1024) / seconds);        System.out.printf("slowest five requests (ms): %s%n", Arrays.toString(worstMs));    }}
The Dockerfile
labs/jvm/gc-pressure/DockerfileDockerfile
# Java 21 lab image. Build:  docker build -t jvm-lab-gc-pressure .FROM eclipse-temurin:21-jdkWORKDIR /labCOPY GcPressure.java .RUN javac -d out GcPressure.java# The JVM reads the container's CPU and memory limits (container support is on by default),# so --cpus and --memory on "docker run" change what Runtime.availableProcessors() and the default heap size report.ENTRYPOINT ["java", "-cp", "out"]CMD ["GcPressure"]

$ java -XX:ActiveProcessorCount=2 -Xmx256m -Xlog:gc GcPressure.java 60

GC logG1, live set 60 MB: steady state

Real JVM output
[2.789s][info][gc] GC(23) Pause Young (Prepare Mixed) (G1 Evacuation Pause) 93M->79M(130M) 1.529ms
[2.934s][info][gc] GC(24) Pause Young (Mixed) (G1 Evacuation Pause) 108M->86M(130M) 1.779ms
[3.057s][info][gc] GC(25) Pause Young (Normal) (G1 Evacuation Pause) 111M->94M(130M) 1.263ms
[3.156s][info][gc] GC(26) Pause Young (Concurrent Start) (G1 Evacuation Pause) 114M->100M(130M) 1.410ms
[3.156s][info][gc] GC(27) Concurrent Mark Cycle
[3.162s][info][gc] GC(27) Pause Remark 101M->75M(130M) 1.622ms
[3.164s][info][gc] GC(27) Pause Cleanup 76M->76M(130M) 0.051ms
[3.164s][info][gc] GC(27) Concurrent Mark Cycle 8.186ms
[3.235s][info][gc] GC(28) Pause Young (Prepare Mixed) (G1 Evacuation Pause) 90M->79M(130M) 1.311ms
[3.379s][info][gc] GC(29) Pause Young (Mixed) (G1 Evacuation Pause) 108M->86M(130M) 1.840ms
[3.503s][info][gc] GC(30) Pause Young (Normal) (G1 Evacuation Pause) 111M->93M(130M) 1.555ms
[3.607s][info][gc] GC(31) Pause Young (Normal) (G1 Evacuation Pause) 114M->100M(130M) 1.472ms
[3.691s][info][gc] GC(32) Pause Young (Normal) (G1 Evacuation Pause) 117M->105M(130M) 1.341ms
[3.760s][info][gc] GC(33) Pause Young (Concurrent Start) (G1 Evacuation Pause) 119M->109M(130M) 1.380ms
[3.760s][info][gc] GC(34) Concurrent Mark Cycle
Each line is uptime, collection number, kind, heap before -> after (committed heap), pause. Young collections come every 100 to 150 ms and take 1 to 2 ms. A Concurrent Start begins marking; Pause Remark then drops the heap by about 25 MB (101M->75M), and a Prepare Mixed and a Mixed collection follow. No Pause Full anywhere in the 133 lines.

$ java -XX:ActiveProcessorCount=2 -Xmx256m -Xlog:gc GcPressure.java 60

Program outputG1, live set 60 MB: what the program measured

Real JVM output
[8.409s][info][gc] GC(88) Pause Young (Normal) (G1 Evacuation Pause) 113M->98M(130M) 0.899ms

live set target 60 MB, max heap 256 MB
requests 7,440 in 8 s, allocated 1,600 MB (200 MB/s)
slowest five requests (ms): [4, 5, 6, 6, 7]
The program times every request. Its five slowest took 4 to 7 ms, which is the length of the longest young pauses early in the run.

$ java -XX:ActiveProcessorCount=2 -Xmx256m -Xlog:gc GcPressure.java 215

GC logG1, live set 215 MB: the first full collection

Real JVM output
[4.354s][info][gc] GC(38) Pause Remark 246M->241M(256M) 1.042ms
[4.356s][info][gc] GC(38) Pause Cleanup 242M->242M(256M) 0.061ms
[4.356s][info][gc] GC(38) Concurrent Mark Cycle 10.857ms
[4.395s][info][gc] GC(39) Pause Young (Prepare Mixed) (G1 Evacuation Pause) 249M->243M(256M) 0.644ms
[4.445s][info][gc] GC(40) Pause Young (Mixed) (G1 Evacuation Pause) (Evacuation Failure) 253M->249M(256M) 0.610ms
[4.480s][info][gc] GC(41) Pause Young (Concurrent Start) (G1 Evacuation Pause) (Evacuation Failure) 255M->255M(256M) 0.881ms
[4.480s][info][gc] GC(43) Concurrent Mark Cycle
[4.501s][info][gc] GC(42) Pause Full (G1 Compaction Pause) 255M->218M(256M) 20.922ms
[4.501s][info][gc] GC(43) Concurrent Mark Cycle 21.024ms
[4.539s][info][gc] GC(44) Pause Young (Normal) (G1 Evacuation Pause) 230M->221M(256M) 0.540ms
[4.594s][info][gc] GC(45) Pause Young (Concurrent Start) (G1 Evacuation Pause) 232M->225M(256M) 1.139ms
Follow the heap-after numbers: 241M, 242M, 243M, 249M, 255M of 256M. Two collections are tagged (Evacuation Failure); the second frees nothing (255M->255M). Then Pause Full (G1 Compaction Pause) 255M->218M takes 20.9 ms. 218 MB is the live set: a full collection cannot go lower.

$ java -XX:ActiveProcessorCount=2 -Xmx256m -Xlog:gc GcPressure.java 215

Program outputG1, live set 215 MB: the end of the run

Real JVM output
[8.383s][info][gc] GC(156) Pause Young (Mixed) (G1 Evacuation Pause) (Evacuation Failure) 240M->236M(256M) 1.014ms
[8.412s][info][gc] GC(157) Pause Young (Concurrent Start) (G1 Evacuation Pause) (Evacuation Failure) 242M->242M(256M) 0.870ms
[8.412s][info][gc] GC(159) Concurrent Mark Cycle
[8.431s][info][gc] GC(158) Pause Full (G1 Compaction Pause) 242M->218M(256M) 19.116ms
[8.431s][info][gc] GC(159) Concurrent Mark Cycle 19.242ms

live set target 215 MB, max heap 256 MB
requests 7,438 in 8 s, allocated 1,600 MB (200 MB/s)
slowest five requests (ms): [18, 19, 20, 21, 22]
The eighth full collection, 19.1 ms, and the program's own measurement: the five slowest requests took 18 to 22 ms. Same code, same allocation rate, same request count as the 60 MB run. Only the live set changed.

$ java -XX:ActiveProcessorCount=2 -Xmx256m -XX:+UseZGC -XX:+ZGenerational -Xlog:gc GcPressure.java 215

GC logGenerational ZGC, live set 215 MB: allocation stalls

Real JVM output
[5.228s][info][gc] GC(62) Major Collection (Allocation Rate)
[5.231s][info][gc] Allocation Stall (main) 2.220ms
[5.232s][info][gc] GC(63) Minor Collection (Allocation Stall)
[5.244s][info][gc] GC(63) Minor Collection (Allocation Stall) 252M(98%)->254M(99%) 0.012s
[5.254s][info][gc] GC(64) Minor Collection (Allocation Stall)
[5.259s][info][gc] Allocation Stall (main) 5.027ms
[5.264s][info][gc] GC(64) Minor Collection (Allocation Stall) 256M(100%)->252M(98%) 0.009s
[5.272s][info][gc] GC(62) Major Collection (Allocation Rate) 256M(100%)->246M(96%) 0.044s
[5.292s][info][gc] GC(65) Major Collection (Allocation Rate)
[5.303s][info][gc] GC(66) Minor Collection (Allocation Rate)
[5.319s][info][gc] Allocation Stall (main) 3.687ms
[5.319s][info][gc] GC(66) Minor Collection (Allocation Rate) 252M(98%)->254M(99%) 0.016s
[5.327s][info][gc] GC(67) Minor Collection (Allocation Rate)
[5.327s][info][gc] Relocation Stall (main) 0.132ms
[5.329s][info][gc] Allocation Stall (main) 1.377ms
[5.331s][info][gc] GC(67) Minor Collection (Allocation Rate) 256M(100%)->248M(97%) 0.005s
[5.331s][info][gc] GC(65) Major Collection (Allocation Rate) 252M(98%)->248M(97%) 0.040s
[5.365s][info][gc] GC(68) Major Collection (Allocation Rate)
[5.369s][info][gc] Allocation Stall (main) 2.214ms
[5.369s][info][gc] GC(69) Minor Collection (Allocation Stall)
[5.378s][info][gc] GC(69) Minor Collection (Allocation Stall) 252M(98%)->252M(98%) 0.009s
[5.390s][info][gc] GC(68) Major Collection (Allocation Rate) 256M(100%)->248M(97%) 0.025s
There is no Pause line: ZGC's collections run next to the application, and each has a start line and an end line with the heap as a percentage. The heap sits at 96 to 100%. Allocation Stall (main) 5.027ms means the main thread waited that long for memory. Collections triggered by a stall are labelled (Allocation Stall).

$ java -XX:ActiveProcessorCount=2 -Xmx256m -XX:+UseZGC -XX:+ZGenerational -Xlog:gc GcPressure.java 60

GC logGenerational ZGC, live set 60 MB

Real JVM output
[0.037s][info][gc] Using The Z Garbage Collector
[0.321s][info][gc] GC(0) Major Collection (Warmup)
[0.391s][info][gc] GC(0) Major Collection (Warmup) 26M(10%)->28M(11%) 0.070s
[0.512s][info][gc] GC(1) Major Collection (Warmup)
[0.546s][info][gc] GC(1) Major Collection (Warmup) 52M(20%)->36M(14%) 0.035s
[0.740s][info][gc] GC(2) Major Collection (Warmup)
[0.764s][info][gc] GC(2) Major Collection (Warmup) 78M(30%)->48M(19%) 0.025s
[1.558s][info][gc] GC(3) Minor Collection (Allocation Rate)
[1.586s][info][gc] GC(3) Minor Collection (Allocation Rate) 216M(84%)->88M(34%) 0.027s
[2.227s][info][gc] GC(4) Minor Collection (Allocation Rate)
[2.237s][info][gc] GC(4) Minor Collection (Allocation Rate) 224M(88%)->84M(33%) 0.010s
[2.911s][info][gc] GC(5) Minor Collection (Allocation Rate)
[2.920s][info][gc] GC(5) Minor Collection (Allocation Rate) 228M(89%)->88M(34%) 0.008s
[3.273s][info][gc] GC(6) Major Collection (Proactive)
[3.297s][info][gc] GC(6) Major Collection (Proactive) 164M(64%)->86M(34%) 0.024s
[3.995s][info][gc] GC(7) Minor Collection (High Usage)
[4.004s][info][gc] GC(7) Minor Collection (High Usage) 234M(91%)->126M(49%) 0.009s
[4.509s][info][gc] GC(8) Minor Collection (High Usage)
[4.515s][info][gc] GC(8) Minor Collection (High Usage) 234M(91%)->128M(50%) 0.007s
[5.012s][info][gc] GC(9) Minor Collection (High Usage)
[5.019s][info][gc] GC(9) Minor Collection (High Usage) 234M(91%)->126M(49%) 0.006s
[5.202s][info][gc] GC(10) Major Collection (Proactive)
[5.227s][info][gc] GC(10) Major Collection (Proactive) 166M(65%)->90M(35%) 0.024s
[5.910s][info][gc] GC(11) Minor Collection (High Usage)
[5.918s][info][gc] GC(11) Minor Collection (High Usage) 234M(91%)->108M(42%) 0.008s
[6.509s][info][gc] GC(12) Minor Collection (High Usage)
[6.516s][info][gc] GC(12) Minor Collection (High Usage) 234M(91%)->110M(43%) 0.007s
[7.005s][info][gc] GC(13) Major Collection (Proactive)
[7.029s][info][gc] GC(13) Major Collection (Proactive) 214M(84%)->86M(34%) 0.024s
[7.729s][info][gc] GC(14) Minor Collection (High Usage)
[7.737s][info][gc] GC(14) Minor Collection (High Usage) 234M(91%)->82M(32%) 0.008s
[8.452s][info][gc] GC(15) Minor Collection (High Usage)
[8.460s][info][gc] GC(15) Minor Collection (High Usage) 234M(91%)->80M(31%) 0.008s

live set target 60 MB, max heap 256 MB
requests 7,435 in 8 s, allocated 1,600 MB (200 MB/s)
slowest five requests (ms): [0, 0, 0, 0, 3]
With room to work in, ZGC lets the heap fill to about 90% and then collects it back to between a third and a half, once or twice a second. No stalls, and the slowest requests took 0 to 3 ms. Note that the heap runs much fuller between collections than under G1.

Real run and simulation: what each one shows

  • The captured files come from running the program directly on OpenJDK 21.0.13 with -XX:ActiveProcessorCount=2, not in Docker. The Docker commands above were not executed when this was written.
  • Real G1 started with 126 MB committed and grew the heap towards 256 MB as the live set grew. The simulation commits the whole heap at once, so its (256M) column never changes and its marking cycles start later.
  • Real numbers at 215 MB: 114 young pauses and 8 full collections of 15 to 21 ms in 8 seconds. The simulation gives full collections of 16 to 19 ms about 1.3 times per second. Close in size, a little low in frequency.
  • Real generational ZGC runs short minor collections and longer major ones side by side, and its stalls were 0.05 to 5 ms. The simulation has a single kind of cycle, so its log lines all say Major Collection and its cycles are longer than the real ones.
  • The real program's slowest requests under ZGC were 2 to 5 ms, against 18 to 22 ms under G1 with the same live set. The simulation shows the same direction, but it lets you push the allocation rate until the stalls are far worse than any G1 pause, which the real run did not try.
  • The simulation shows the heap between collections and lets you change the live set while it runs. A GC log only gives you the heap at each collection.

How you meet this in production

  • The first symptom is tail latency, not errors: p99 or p99.9 rises in regular spikes while the median and CPU stay where they were. Throughput may drop a few percent.
  • Turn on -Xlog:gc* (to a rotating file) and search it for Pause Full and Evacuation Failure. Then read the heap-after value of the full collections: that is the live set. If it is most of -Xmx, no tuning flag will help.
  • Spring Boot Actuator publishes garbage collection and memory metrics under the jvm. prefix. Graph pause time and the heap after collection next to request latency: the spikes line up.
  • A JFR recording (jcmd <pid> JFR.start duration=60s filename=gc.jfr) holds every pause with its cause and the allocation rate, and can be opened after the incident.
  • Practise the diagnosis on Spikes every few seconds.
Investigate · Practicep99 latency is 20 ms on an endpoint that should take 2 ms

References

References: Java SE 21 GC Tuning Guide: Garbage-First (G1) Garbage Collector ↗ · Java SE 21 GC Tuning Guide: Garbage-First Garbage Collector Tuning (Full GC, evacuation failure) ↗ · Java SE 21 GC Tuning Guide: Available Collectors (G1, ZGC, how to select) ↗ · Java SE 21 Tool Specifications: the jcmd command ↗ · Report a problem