Path 2 · lesson 2 of 4
GC tuning and pause analysis
Read a GC log line by line, name the cause of a long pause or a stall, and tune in the order that works: heap size, live set, allocation rate, then the pause target.
What you will learn
- Read every field of a JDK 21 G1 log line and of the detail lines
-Xlog:gc*adds - Work out the allocation rate and the live set of a service from its GC log
- Tell young, mixed and full pauses apart and name what makes each one long
- Recognise evacuation failure, humongous allocation and ZGC allocation stalls in a log
- Apply tuning in order: heap size, live set, allocation rate, pause target, and leave the rest alone
- Diagnose periodic latency spikes by lining them up with pauses, safepoints or stalls
Before you start
The concept
GC tuning is mostly arithmetic on two measurements. The allocation rate (MB of new objects per second) decides how often the collector has to run. The live set (MB still reachable after a collection) decides how much work each run is and how much of the heap is left over as working room. A collector in trouble is nearly always one where the live set plus the room that the allocation rate needs is larger than the heap.
The order of tuning follows from that. First make the heap fit the live set and the allocation rate. If you cannot, shrink the live set or the allocation rate in the code. Only on a heap that fits does the pause target mean anything, and only after that are the fine-grained flags worth a look. Most services stop at step one.
| Step | Question | Where the answer is |
|---|---|---|
| 1. Measure | What do the pauses look like, and of which kind are they? | -Xlog:gc* written to a file from the first day |
| 2. Heap size | Does the live set leave enough room? | Heap used after a full collection or after the mixed collections, against -Xmx |
| 3. Live set | Is all that retained data needed? | A class histogram or a heap dump (next lesson) |
| 4. Allocation rate | What allocates, and does it have to? | JFR allocation events or an allocation profile (lesson on JFR and async-profiler) |
| 5. Pause target | On a healthy heap, are the pauses still too long? | -XX:MaxGCPauseMillis, or a move to ZGC |
Why it exists
The usual report is not "GC is slow". It is "p99 latency jumps every few seconds and nothing in the application logs explains it". The request that was slow did nothing unusual: it was stopped, together with every other thread, while the collector worked, or it waited for memory. The application's own timers cannot see why, because the code that would log it was stopped too.
The second reason to learn this is the amount of bad advice. Lists of twenty G1 flags are easy to find and easy to paste into a Dockerfile. The experiment below takes the lab's worst case, a 215 MB live set in a 256 MB heap, and tries three of the usual answers.
Program outputFlags against a heap that is too small, then a heap that is not (GcPressure.java 215, 8 seconds each)
Real JVM output$ java -XX:ActiveProcessorCount=2 -Xmx256m -Xlog:gc:file=gc.log GcPressure.java 215 live set target 215 MB, max heap 256 MB $ grep -c 'Pause Young' gc.log; grep -c 'Evacuation Failure' gc.log; grep -c 'Pause Full' gc.log 114 34 13 $ java -XX:ActiveProcessorCount=2 -Xmx256m -XX:MaxGCPauseMillis=5 -Xlog:gc:file=gc.log GcPressure.java 215 live set target 215 MB, max heap 256 MB $ grep -c 'Pause Young' gc.log; grep -c 'Evacuation Failure' gc.log; grep -c 'Pause Full' gc.log 171 36 14 $ java -XX:ActiveProcessorCount=2 -Xmx256m -Xmn96m -Xlog:gc:file=gc.log GcPressure.java 215 live set target 215 MB, max heap 256 MB $ grep -c 'Pause Young' gc.log; grep -c 'Evacuation Failure' gc.log; grep -c 'Pause Full' gc.log 92 47 18 $ java -XX:ActiveProcessorCount=2 -Xmx512m -Xlog:gc:file=gc.log GcPressure.java 215 live set target 215 MB, max heap 512 MB $ grep -c 'Pause Young' gc.log; grep -c 'Evacuation Failure' gc.log; grep -c 'Pause Full' gc.log 30 0 0
How it works inside
Turning the log on
JDK 21 logs through unified logging. The option has four parts separated by colons: -Xlog:<what>:<output>:<decorators>:<output-options>. what is a list of tag sets with an optional level, and a trailing * matches every tag set that contains those tags. -Xlog:gc prints one summary line per collection. -Xlog:gc* adds every other set that includes gc: gc,start, gc,phases, gc,heap, gc,cpu, gc,init and more. Adding safepoint logs every stop of the application, whatever caused it.
| Part of the line | Example | Meaning |
|---|---|---|
| Uptime | [4.683s] | Seconds since the JVM started. Add time to the decorators for wall-clock time, which is what you need to match application logs. |
| Level | [info] | The level the message was logged at. |
| Tags | [gc,phases] | The tag set. [gc] alone is the summary line. |
| Collection id | GC(47) | A counter. Every line of one collection carries the same id, so grep 'GC(47)' extracts it. |
| Kind | Pause Young (Prepare Mixed) | Pause means the application was stopped. Concurrent means it was not. |
| Cause | (G1 Evacuation Pause) | Why it started. G1 Evacuation Pause is the normal reason: eden was full. Others: G1 Humongous Allocation, G1 Compaction Pause, System.gc(), Heap Dump Initiated GC. |
| Heap | 249M->247M(256M) | Used before, used after, and committed heap in brackets. |
| Duration | 0.757ms | For a Pause line, how long the application was stopped for the collection. |
[4.683s][info][gc] GC(47) Pause Young (Prepare Mixed) (G1 Evacuation Pause) 249M->247M(256M) 0.757ms.One young collection, line by line
GC logA young collection with -Xlog:gc*,safepoint (G1, live set 215 MB, -Xmx256m)
Real JVM output[4.683s][info][gc,start ] GC(47) Pause Young (Prepare Mixed) (G1 Evacuation Pause) [4.683s][info][gc,task ] GC(47) Using 2 workers of 2 for evacuation [4.683s][info][gc,phases ] GC(47) Pre Evacuate Collection Set: 0.05ms [4.683s][info][gc,phases ] GC(47) Merge Heap Roots: 0.02ms [4.683s][info][gc,phases ] GC(47) Evacuate Collection Set: 0.49ms [4.683s][info][gc,phases ] GC(47) Post Evacuate Collection Set: 0.11ms [4.683s][info][gc,phases ] GC(47) Other: 0.03ms [4.683s][info][gc,heap ] GC(47) Eden regions: 3->0(8) [4.683s][info][gc,heap ] GC(47) Survivor regions: 1->1(1) [4.683s][info][gc,heap ] GC(47) Old regions: 246->247 [4.683s][info][gc,heap ] GC(47) Humongous regions: 0->0 [4.683s][info][gc,metaspace] GC(47) Metaspace: 10070K(10240K)->10070K(10240K) NonClass: 8884K(8960K)->8884K(8960K) Class: 1185K(1280K)->1185K(1280K) [4.683s][info][gc ] GC(47) Pause Young (Prepare Mixed) (G1 Evacuation Pause) 249M->247M(256M) 0.757ms [4.683s][info][gc,cpu ] GC(47) User=0.00s Sys=0.00s Real=0.00s [4.683s][info][safepoint ] Safepoint "G1CollectForAllocation", Time since last: 1598107 ns, Reaching safepoint: 441 ns, Cleanup: 441 ns, At safepoint: 770166 ns, Leaving safepoint: 3845 ns, Total: 774893 ns
gc,task: both GC threads worked. gc,phases: the pause split into its parts; Evacuate Collection Set is the copying of live objects and is normally the largest. gc,heap: regions per kind, before and after, with the next target in brackets. Here eden had only 3 regions to give and the old generation holds 247 of the heap's 256: that single line says the heap is full of long-lived data. gc,cpu: user, system and wall-clock time. The safepoint line is the stop as the application saw it: 0.775 ms in total, of which the collection itself was 0.770 ms.- Evacuate Collection Set large means much live data was copied. The young generation is big, or objects live just long enough to be copied several times before they die.
- Merge Heap Roots large means many references from old regions into the regions being collected (large remembered sets).
Syshigh ingc,cpu, orRealfar aboveUserdivided by the worker count, points outside the JVM: memory being paged in, transparent huge pages, a log on a slow disk, or a machine with no CPU to spare. The G1 tuning guide lists exactly these.Old regionsrising collection after collection with nothing bringing it down is promotion. Whether it is a leak or a normal fill before the next mixed collections is the question for the next lesson.
Allocation rate and live set from the log
Two consecutive young collections give the allocation rate: what was allocated between them is the before of the second minus the after of the first. From the comfortable run in the previous lesson: GC(87) ended at 8.300 s with 91M used, and GC(88) started at 8.409 s with 113M. That is 22 MB in 0.109 s, about 200 MB/s, which is the rate the lab program was set to. The live set is the after value of a full collection, or the lowest after value reached at the end of a round of mixed collections. In the pressure run every full collection ends at 218M: 215 MB of session data plus a few megabytes of everything else.
Pause kinds and what makes each one long
| Pause | Normal length | It is long when | First response |
|---|---|---|---|
Pause Young (Normal) | Milliseconds, up to the pause goal | Many objects survive: a large young generation with a high survival rate | Look at what survives and why. Do not fix the young size by hand |
Pause Young (Mixed) | Similar to young, slightly longer | The old regions chosen hold much live data or have large remembered sets | Usually nothing. If they exceed the goal on a healthy heap, see the G1 tuning guide on mixed collections |
Pause Remark | A few milliseconds | Very many soft, weak or phantom references, or much class unloading | Check gc+phases for reference processing |
Pause Young (...) (Evacuation Failure) | Any | There was no free region to copy survivors into | A sizing problem. Expect Pause Full next |
Pause Full (G1 Compaction Pause) | Tens of ms on a small heap, seconds on a large one | Always: it marks and compacts the whole heap while everything is stopped | More heap, a smaller live set, or fewer humongous objects |
Pause Full (System.gc()) | As above | Code or a tool called System.gc() | Find the caller. -XX:+ExplicitGCInvokesConcurrent turns it into a concurrent cycle |
Evacuation failure and the full collection that follows
A young or mixed collection copies survivors into free regions. If there are none left, the collection cannot complete normally: JDK 21 tags the line (Evacuation Failure). Logs from older JDKs describe the same event as to-space exhaustion. The objects that could not be moved stay where they are, almost nothing is freed, and G1's last resort is a full collection.
GC logEvacuation failure, then a full collection (the lines that follow GC(47) above)
Real JVM output[4.722s][info][gc,heap ] GC(48) Eden regions: 8->0(1) [4.722s][info][gc,heap ] GC(48) Survivor regions: 1->0(0) [4.723s][info][gc,heap ] GC(48) Old regions: 247->256 [4.723s][info][gc,heap ] GC(48) Humongous regions: 0->0 [4.723s][info][gc ] GC(48) Pause Young (Mixed) (G1 Evacuation Pause) (Evacuation Failure) 255M->255M(256M) 0.853ms [4.723s][info][gc,ergo ] Attempting full compaction [4.723s][info][gc,start ] GC(49) Pause Full (G1 Compaction Pause) [4.723s][info][gc,task ] GC(49) Using 2 workers of 2 for full compaction [4.727s][info][gc,phases ] GC(49) Phase 1: Mark live objects 4.738ms [4.729s][info][gc,phases ] GC(49) Phase 2: Prepare compaction 1.143ms [4.730s][info][gc,phases ] GC(49) Phase 3: Adjust pointers 1.662ms [4.741s][info][gc,phases ] GC(49) Phase 4: Compact heap 10.576ms [4.741s][info][gc,phases ] GC(49) Phase 5: Reset Metadata 0.381ms [4.742s][info][gc,heap ] GC(49) Old regions: 256->233 [4.742s][info][gc ] GC(49) Pause Full (G1 Compaction Pause) 255M->218M(256M) 19.569ms [4.742s][info][gc,cpu ] GC(49) User=0.03s Sys=0.00s Real=0.02s [4.742s][info][safepoint ] Safepoint "G1CollectForAllocation", Time since last: 38230137 ns, Reaching safepoint: 1672 ns, Cleanup: 2033 ns, At safepoint: 20464504 ns, Leaving safepoint: 7942 ns, Total: 20476151 ns
Old regions: 247->256: after GC(48) all 256 regions of the heap are old, and 255M->255M shows it freed nothing. Attempting full compaction is G1 announcing the fallback. The full collection takes 19.6 ms on this 256 MB heap, more than half of it in Phase 4: Compact heap, and brings the heap to 218M: the live set. The safepoint line reports one stop of 20.5 ms, because the failed young collection and the full collection happen in the same stop. A request in flight waited for all of it.Safepoints: stops the GC lines do not explain
The JVM can only stop a thread at a safepoint, a place in the code where the thread's state is fully known. To pause the application, the JVM asks every thread to stop at its next safepoint and waits until all of them have. The safepoint log line splits the stop into the time spent reaching the safepoint, the time at it doing the operation, and the total. Two things follow. A thread that takes long to reach a safepoint makes every pause longer for everyone, and the GC duration will not show it. And garbage collection is not the only operation that needs a safepoint: thread dumps, heap dumps, class redefinition and some deoptimisations do as well. If latency spikes do not line up with any Pause line, log safepoint and look at the operation name and at Reaching safepoint.
Humongous allocation
A collection whose cause is G1 Humongous Allocation was started by a single large array, not by eden filling up. The previous lesson shows the log. For tuning it matters because it changes the arithmetic: humongous objects go straight to the old generation, so they raise old occupancy at the allocation rate of the large arrays, start marking cycles early, and need contiguous regions. -Xlog:gc+heap prints Humongous regions: X->Y for every collection; a high X next to a low Y is a stream of short-lived large arrays.
ZGC: allocation stalls instead of pauses
GC logGenerational ZGC under the same pressure (-XX:+UseZGC -XX:+ZGenerational -Xlog:gc GcPressure.java 215)
Real JVM output[4.181s][info][gc] GC(23) Minor Collection (Allocation Rate) [4.188s][info][gc] GC(23) Minor Collection (Allocation Rate) 248M(97%)->248M(97%) 0.007s [4.209s][info][gc] GC(24) Major Collection (Allocation Rate) ... [5.231s][info][gc] Allocation Stall (main) 2.220ms [5.232s][info][gc] GC(63) Minor Collection (Allocation Stall) [5.244s][info][gc] GC(63) Minor Collection (Allocation Stall) 252M(98%)->254M(99%) 0.012s [5.254s][info][gc] GC(64) Minor Collection (Allocation Stall) [5.259s][info][gc] Allocation Stall (main) 5.027ms [5.264s][info][gc] GC(64) Minor Collection (Allocation Stall) 256M(100%)->252M(98%) 0.009s
248M(97%)->248M(97%): a collection that reclaimed nothing. Then the cause in brackets changes from Allocation Rate (ZGC predicting it must start now to keep up) to Allocation Stall (it did not keep up). And the Allocation Stall (main) 5.027ms lines themselves: the thread named in brackets waited that long for memory. There is no Pause anywhere, so pause-time dashboards stay flat while requests are delayed. Under -Xlog:gc* every collection also prints an Allocation Stalls: row, which should be all zeros.What to change, and what to leave alone
- Heap size. Compare the live set with
-Xmx. The lab's 215 MB live set failed in 256 MB and ran without a single full collection in 512 MB. As a starting point, give G1 a heap of two to three times the live set and confirm in the log; ZGC wants more. Set-Xmsequal to-Xmxso the heap is not resized under load. - Live set. If the heap cannot grow, find what is retained: caches without a bound, session data, large in-memory indexes. A cache limited by entry count is not limited in bytes.
- Allocation rate. Fewer allocations mean fewer collections and fewer objects that survive by accident of timing. Find the allocating code with JFR before changing anything.
- Pause target. Only now. Lower
-XX:MaxGCPauseMillisif pauses on a healthy heap are too long for the service, and accept more frequent collections. If the goal is single-digit milliseconds, change collector instead. - Specific findings. Humongous objects: remove them or raise
-XX:G1HeapRegionSize. Marking that finishes too late: more heap first, then-XX:G1ReservePercentor more-XX:ConcGCThreads, as the G1 tuning guide describes.
Architecture
Drawing diagram…
Flowchart for diagnosing periodic latency spikes. If spike times match pause lines, the kind of pause decides: full collections or evacuation failure mean the heap is too small, humongous allocation means large arrays, long young or mixed pauses lead to checking system time and then live data. If they do not match pauses, check safepoint lines and allocation stalls, and otherwise look outside the collector.
flowchart TD
A["Periodic latency spikes"] --> B{"Do spike times match Pause lines in the GC log"}
B -->|"Yes"| C{"Which kind of pause"}
B -->|"No"| D{"Do they match safepoint lines or Allocation Stall lines"}
C -->|"Pause Full or Evacuation Failure"| E["Heap too small for live set and allocation rate"]
C -->|"Humongous Allocation"| F["Large arrays, remove them or raise region size"]
C -->|"Young or Mixed above the goal"| G{"Is Sys time high or Real far above User"}
G -->|"Yes"| H["Environment, memory paging, huge pages, CPU throttling"]
G -->|"No"| I["Much live data copied, lower pause target or move to ZGC"]
D -->|"Safepoint with long Reaching time"| J["A thread is slow to stop, find it"]
D -->|"Allocation Stall"| K["ZGC out of headroom, more heap or smaller live set"]
D -->|"Neither"| L["Not GC, look at locks, pools and downstream calls"]- The first step is a join on time. Take the timestamps of the slow requests from access logs or traces and the timestamps of
Pause,safepointandAllocation Stalllines. This is why the log needs thetimedecoration: uptime alone cannot be matched to anything. - If the spikes line up with pauses, the kind of pause carries the diagnosis.
Pause Fulland(Evacuation Failure)are a sizing problem and no pause-target setting will remove them, as the experiment showed. - Long pauses of an ordinary kind split into two cases by the
gc,cpuline. High system time, or wall-clock time far above CPU time, is the environment. Otherwise the pause was long because it had much to copy. - If nothing in the GC log matches, believe it. The same symptom also comes from an exhausted thread pool, a contended lock or a slow dependency, and a GC log can rule the collector out in a few minutes. The Investigate case on latency spikes practises this.
Step by step
Drawing diagram…
Sequence diagram. A request thread fails to allocate because eden is full and asks the VM thread for a collection. The VM thread stops all threads at a safepoint, GC workers evacuate live objects, and the threads are resumed. The client receives its response later by the length of the pause.
sequenceDiagram participant C as Client participant R as Request thread participant V as VM thread participant W as GC workers C->>R: POST /checkout R->>R: Allocates, eden has no room R->>V: Request a collection V->>R: Stop at the next safepoint Note over R,V: Reaching safepoint, every thread must arrive V->>W: Evacuate live objects W-->>V: Done after 20 ms V->>R: Resume all threads R->>R: Allocation succeeds, handler continues R-->>C: 200 OK, 20 ms later than usual
- The request thread that triggered the collection and every other application thread wait for the same time. A pause of 20 ms adds 20 ms to each request in flight, not to one.
Reaching safepointis the gap between the VM thread asking and the last thread arriving. It is normally microseconds. When it is not, the GC duration looks fine and the total stop does not.- The handler's own timing brackets the pause without seeing it. A span or a log line shows 20 ms more than usual between two statements that normally take microseconds: that pattern, scattered across unrelated endpoints at the same instant, is the signature of a stop-the-world event.
In a real system
A session service keeps recent customer sessions in memory to avoid reloading them from the database. It runs with -Xmx256m in a small container. After a marketing campaign the number of active sessions triples. Dashboards show p99 latency jumping to five times its usual value several times a second, CPU a little higher, no errors. The service has a GC pause listener (the one in the implementation section below) and its output is the first thing the on-call engineer reads.
Program outputCollections of 10 ms or more, reported from inside the process (GcTuningPauseAnalysis.java 215 10)
Real JVM outputslow gc: G1 Young Generation end of minor GC cause=G1 Evacuation Pause 12 ms heap 249 MB -> 250 MB slow gc: G1 Old Generation end of major GC cause=G1 Compaction Pause 76 ms heap 255 MB -> 218 MB slow gc: G1 Young Generation end of minor GC cause=G1 Evacuation Pause 16 ms heap 239 MB -> 240 MB slow gc: G1 Old Generation end of major GC cause=G1 Compaction Pause 51 ms heap 241 MB -> 218 MB slow gc: G1 Old Generation end of major GC cause=G1 Compaction Pause 82 ms heap 240 MB -> 218 MB slow gc: G1 Young Generation end of minor GC cause=G1 Evacuation Pause 10 ms heap 229 MB -> 230 MB slow gc: G1 Old Generation end of major GC cause=G1 Compaction Pause 47 ms heap 241 MB -> 218 MB ... collections 53, total collection time 848 ms in 6 s, allocated 1,199 MB
G1 Old Generation with cause G1 Compaction Pause is a full collection. Each one ends at 218 MB: that is the live set, 85% of a 256 MB heap. Young collections free nothing (249 MB -> 250 MB). The last line gives the GC overhead: 848 ms of collection in 6 seconds, 14% of the time.- Match the times. The slow requests cluster at the timestamps of the
Compaction Pauselines. The spikes are full collections. - Read the live set. 218 MB after every full collection, in a 256 MB heap. There are about 38 MB of working room for an allocation rate of 200 MB/s: less than a fifth of a second.
- Decide where the room comes from. Either the heap grows, or the live set shrinks. The session window is the live set, so both are possible: raise
-Xmxand the container limit, or cap the cache by bytes. - Do not reach for the pause target. The experiment at the top of this lesson is this exact case:
-XX:MaxGCPauseMillis=5changed nothing about the full collections. - Verify with the same evidence. After the change, the log should show no
Pause Full, no(Evacuation Failure), andaftervalues well below the heap size.
On a Spring Boot service the same numbers are available without shell access. Micrometer's JVM metrics, which Spring Boot Actuator registers, include the timer jvm.gc.pause (tagged with the cause and the action), jvm.gc.live.data.size, jvm.gc.max.data.size and the counter jvm.gc.memory.allocated. Live data size against max data size is the live-set ratio; the rate of the allocated counter is the allocation rate. GET /actuator/metrics/jvm.gc.pause shows the first of them.
Java and Spring Boot
Two things every service should have before it needs them: a GC log with wall-clock timestamps, and a line in the application log whenever a collection takes longer than the latency budget allows. The listener uses the JDK's own notification API. Its logic is the lab program GcTuningPauseAnalysis.java, which was compiled and run on OpenJDK 21.0.13 to produce the output above; the Spring wrapper around it was not compiled against Spring here.
JAVA_TOOL_OPTIONS="-Xms768m -Xmx768m \ -Xlog:gc*,safepoint:file=/var/log/sessions/gc.log:time,uptime,level,tags:filecount=5,filesize=20m"Why it is written this way
gc*for every GC message at info level,safepointfor every stop of the application with its cause and the time to reach it.timeadds an ISO-8601 timestamp to each line. Without it you cannot match a pause to a slow request in another log.filecount=5,filesize=20mrotates the file. Unified logging rotates file output by default (5 files of 20 MB); stating it makes the disk budget explicit.- Logging can also be switched on in a running JVM:
jcmd <pid> VM.log what=gc*,safepoint output=file=/tmp/gc.log decorators=time,uptime,level,tags.
package com.example.sessions.ops;import java.lang.management.GarbageCollectorMXBean;import java.lang.management.ManagementFactory;import java.lang.management.MemoryPoolMXBean;import java.lang.management.MemoryType;import java.lang.management.MemoryUsage;import java.util.Map;import java.util.Set;import java.util.stream.Collectors;import javax.management.NotificationEmitter;import javax.management.openmbean.CompositeData;import com.sun.management.GarbageCollectionNotificationInfo;import com.sun.management.GcInfo;import jakarta.annotation.PostConstruct;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.stereotype.Component;/** Writes one application log line for every collection that takes longer than the latency budget allows. */@Componentpublic class GcPauseMonitor { private static final Logger log = LoggerFactory.getLogger(GcPauseMonitor.class); private static final long THRESHOLD_MS = 50; @PostConstruct void register() { Set<String> heapPools = ManagementFactory.getMemoryPoolMXBeans().stream() .filter(pool -> pool.getType() == MemoryType.HEAP) .map(MemoryPoolMXBean::getName) .collect(Collectors.toSet()); for (GarbageCollectorMXBean gc : ManagementFactory.getGarbageCollectorMXBeans()) { if (!(gc instanceof NotificationEmitter emitter)) continue; emitter.addNotificationListener((notification, handback) -> { if (!GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION.equals(notification.getType())) return; var info = GarbageCollectionNotificationInfo.from((CompositeData) notification.getUserData()); GcInfo detail = info.getGcInfo(); if (detail.getDuration() < THRESHOLD_MS) return; log.warn("slow gc: collector={} action={} cause={} durationMs={} heapBeforeMb={} heapAfterMb={}", info.getGcName(), info.getGcAction(), info.getGcCause(), detail.getDuration(), heapMb(detail.getMemoryUsageBeforeGc(), heapPools), heapMb(detail.getMemoryUsageAfterGc(), heapPools)); }, null, null); } } private static long heapMb(Map<String, MemoryUsage> pools, Set<String> heapPools) { return pools.entrySet().stream() .filter(e -> heapPools.contains(e.getKey())) .mapToLong(e -> e.getValue().getUsed()) .sum() / (1024 * 1024); }}Why it is written this way
- The JVM sends a notification after every collection.
GcInfocarries its duration and the usage of every memory pool before and after. - The usage maps include Metaspace and the code cache. Summing only the pools of type
HEAPgives numbers that match the GC log. - On G1 the collector name tells you the kind:
G1 Young Generationfor young and mixed collections,G1 Old Generationfor full collections,G1 Concurrent GCfor the Remark and Cleanup pauses. - On ZGC the
Cyclesbeans report the duration of a concurrent cycle, not a pause. A threshold on them would fire constantly and mean nothing. For ZGC, alert on allocation stalls in the log or in JFR instead. - The listener runs on a JVM service thread after the collection. Keep it to a log line or a metric; never allocate heavily or block in it.
Wrong, then right
The session service from the scenario. Sessions are about 64 kB each once the basket and the recommendations are attached. The cache was sized when a 256 MB heap held a few hundred of them, and its limit is a number of entries that somebody raised to 3,500 before the campaign.
Incorrect implementation
package com.example.sessions;import java.util.LinkedHashMap;import java.util.Map;import java.util.function.Function;import org.springframework.stereotype.Component;@Componentpublic class SessionCache { // "3,500 sessions" says nothing about memory. At 64 kB each this is 215 MB of live data. private static final int MAX_ENTRIES = 3_500; private final Map<String, byte[]> sessions = new LinkedHashMap<>(16, 0.75f, true) { @Override protected boolean removeEldestEntry(Map.Entry<String, byte[]> eldest) { return size() > MAX_ENTRIES; } }; public synchronized byte[] get(String sessionId, Function<String, byte[]> loader) { return sessions.computeIfAbsent(sessionId, loader); }}Why it is written this way
- The bound is real: the map never holds more than 3,500 entries, so this is not a leak and the service never throws
OutOfMemoryError. - Nothing connects the bound to the heap. Raising the entry count, or the size of one session, changes the live set without any setting that mentions memory.
Why it fails
- 3,500 entries of 64 kB are 215 MB of objects that are reachable at all times. In a 256 MB heap that leaves about 40 MB for everything else, including the space the collector needs to copy survivors into.
- At 200 MB/s of allocation that room lasts a fraction of a second. Young collections find no free region for survivors and are logged as
(Evacuation Failure); G1 falls back toPause Full. The real run recorded 8 full collections of 15 to 21 ms in 8 seconds, and the slowest requests took 18 to 22 ms against 4 to 7 ms with a 60 MB live set. - Every full collection ends at the same heap usage, because everything it finds is alive. The collector is doing its most expensive work to free a few megabytes, several times a second.
- In production this is p99 latency several times the median with a sawtooth of period well under a second, GC CPU rising with traffic, and no error anywhere. A pause-target flag does not help: the heap is full of data the application says it needs.
Corrected implementation
package com.example.sessions;import java.util.Iterator;import java.util.LinkedHashMap;import java.util.Map;import java.util.function.Function;import org.springframework.stereotype.Component;@Componentpublic class SessionCache { /** The cache may hold at most a quarter of the heap, whatever the heap is and whatever a session weighs. */ private final long maxBytes = Runtime.getRuntime().maxMemory() / 4; private final Map<String, byte[]> sessions = new LinkedHashMap<>(16, 0.75f, true); private long currentBytes; public synchronized byte[] get(String sessionId, Function<String, byte[]> loader) { byte[] cached = sessions.get(sessionId); // access order: this marks the entry as recently used if (cached != null) return cached; byte[] loaded = loader.apply(sessionId); sessions.put(sessionId, loaded); currentBytes += loaded.length; Iterator<Map.Entry<String, byte[]>> eldest = sessions.entrySet().iterator(); while (currentBytes > maxBytes && eldest.hasNext()) { Map.Entry<String, byte[]> entry = eldest.next(); if (entry.getValue() == loaded) break; // never evict what was just loaded currentBytes -= entry.getValue().length; eldest.remove(); } return loaded; } public synchronized long bytesHeld() { return currentBytes; }}Why it is written this way
- The limit is expressed in the unit that matters to the collector: bytes, as a share of
Runtime.maxMemory(). With-Xmx256mit is 64 MB, close to the lab's comfortable 60 MB run; with a larger heap it grows by itself. - The
LinkedHashMapis in access order (third constructor argumenttrue), so iteration starts at the least recently used entry and eviction removes those first. bytesHeld()exists to be published as a gauge. The live set this cache contributes should be on a dashboard, next to heap used after GC.- Compiled on JDK 21 against a stand-in for the Spring annotation. The numbers quoted for both variants come from the lab program, which models the same live sets, not from running this class.
Trade-offs
- A quarter of the heap holds about a thousand sessions instead of 3,500. More requests miss the cache and reload from the database: the cost moved from GC pauses to database load, and that trade has to be measured.
- If the hit rate matters more, the other fix is a larger heap with the cache share unchanged: the real run with
-Xmx512mand the same 215 MB of live data had no full collections. That costs memory on every instance. - The size of a
byte[]is known; the retained size of a real session object graph is not. Estimating it is approximate, and the bound is only as good as the estimate. - One lock around the whole map serialises every lookup. Under high concurrency use a cache library with a weight-based bound instead of this class; the principle, a limit in bytes tied to the heap, stays the same.
Configuration
The settings involved in pause analysis, with the defaults of OpenJDK 21 as printed by java -XX:+PrintFlagsFinal -version on 21.0.13 and documented in the java command reference. The list is short on purpose.
-Xms / -XmxDefault Ergonomic: initial 1/64 and maximum 1/4 of available memory
- What it does
- Initial and maximum heap size.
- When to change it
- The first and usually the only tuning step. Size from the measured live set and allocation rate; set both to the same value for a service.
-Xlog:gc*Default Not enabled (only warnings and errors are logged, to stdout)
- What it does
- Log every message whose tag set contains
gc, at info level. - When to change it
- Always on in production, to a rotated file with the
timedecoration. The overhead is a few lines per collection.
-Xlog:safepointDefault Not enabled
- What it does
- One line per stop of the application: the operation, time to reach the safepoint, time at it, total.
- When to change it
- Add it next to
gc*. It is the only place a long time-to-safepoint or a non-GC stop shows up.
-XX:MaxGCPauseMillisDefault 200
- What it does
- G1's soft pause-time goal in milliseconds.
- When to change it
- Step four, not step one. Lower it on a healthy heap if pauses exceed the service's budget; raise it for batch work that prefers throughput.
-XX:G1ReservePercentDefault 10
- What it does
- Share of the heap G1 keeps free as a reserve, to reduce the chance of evacuation failure.
- When to change it
- The G1 tuning guide names it as a way to make marking start earlier. Raise it only after more heap has been ruled out, and remember that the reserve is heap you pay for.
-XX:ConcGCThreadsDefault Ergonomic, about a quarter of the parallel GC threads (1 on the 2-CPU lab machine)
- What it does
- Threads used for concurrent marking.
- When to change it
- Raise it if marking cycles regularly finish too late and CPU is available. It takes CPU from the application while marking runs.
-XX:+AlwaysPreTouchDefault Disabled
- What it does
- Touch every heap page at startup instead of on first use.
- When to change it
- Use with
-Xmsequal to-Xmxwhengc,cpushows high system time in early pauses. Startup gets slower in exchange.
-Xmn, -XX:NewRatioDefault Not set: G1 sizes the young generation itself
- What it does
- Fix the size of the young generation.
- When to change it
- Do not set them with G1. The tuning guide: setting the young generation size to a single value overrides and practically disables pause-time control.
-XX:+ExplicitGCInvokesConcurrentDefault Disabled
- What it does
- Make
System.gc()start a concurrent cycle instead of a full collection. - When to change it
- For services where a library or tool calls
System.gc()and you seePause Full (System.gc()). Prefer it to-XX:+DisableExplicitGC, which silently ignores the calls.
Hands-on exercise
Find the room, not the flag
Start from a heap with a live set that nearly fills it, read the kind of collections and delays it produces, and bring it back to health by changing heap size, live set and allocation rate one at a time.
Steps · 0 of 6 done
A 215 MB live set in a 256 MB heap
Simulation · not a real JVMThe heap starts under pressure. Change heap size, live set and allocation rate one at a time and watch which kind of collection runs and how long requests are delayed.
What you should see
At the start, young collections fail to free space and full collections recur several times a second, as in the real run (8 full collections of 15 to 21 ms in 8 seconds). The room is about 40 MB, a fifth of a second of allocation. More heap or a smaller live set both remove the full collections once the room is a comfortable multiple of what is allocated between collections; the real run needed no full collection at 512 MB. A lower allocation rate spaces the full collections out but does not remove them while the live set stays at 215 MB. ZGC replaces full collections with allocation stalls and needs more heap again before they stop.
What can go wrong
Lowering the pause target on a heap that is too small
- Symptom
- After setting
-XX:MaxGCPauseMillisto a small value, there are more collections, the same number ofPause Fulllines and no better p99. - Cause
- The goal only steers how much G1 collects per pause. It cannot create free regions. A smaller young generation fills sooner, so collections multiply.
- Fix
- Check the live set against
-Xmxfirst. Remove the full collections with heap or a smaller live set, then decide whether the pause goal needs changing at all.
A GC log without wall-clock time
- Symptom
- The log shows pauses at
[4.742s]and nobody can say which requests they hit. - Cause
- The default decorations are uptime, level and tags. Uptime cannot be joined to access logs or traces.
- Fix
- Add
timeto the decorators:-Xlog:gc*,safepoint:file=gc.log:time,uptime,level,tags.
Trusting the pause duration when the stop was longer
- Symptom
- Requests stall for 300 ms, and the longest GC pause in the log is 20 ms.
- Cause
- The GC duration excludes the time threads took to reach the safepoint, and other VM operations stop the application too.
- Fix
- Enable
-Xlog:safepointand readReaching safepointandTotalon the line for that moment, and the operation name in quotes.
Watching only pause metrics on ZGC
- Symptom
- Dashboards show sub-millisecond pauses while requests are delayed by several milliseconds at a time under load.
- Cause
- Allocation stalls are waits for memory, not pauses. They are logged as
Allocation Stall (<thread>)and counted separately. - Fix
- Alert on
Allocation Stallin the GC log and on theAllocation Stalls:row under-Xlog:gc*. Treat any stall as a heap sizing finding.
Carrying a flag list from an older JDK
- Symptom
- Startup warnings about ignored or obsolete options, a young generation that never changes size, pause goals that are not met.
- Cause
- Options written for CMS or JDK 8 either no longer exist or fight G1's own sizing.
- Fix
- Start from no GC flags except heap size and logging, as the G1 tuning guide recommends, and add a flag only for a finding you can point to in the log.
How to troubleshoot
p99 latency jumps several times a second
Check
grep -E 'Pause Full|Evacuation Failure' gc.log | tail -20
- How to read it
- Any
Pause Full (G1 Compaction Pause)preceded by(Evacuation Failure)lines means the heap has no room. Read theaftervalue of the full collections: it is the live set. Compare it with-Xmx.
You need the live set and heap layout of a running service
Check
jcmd <pid> GC.heap_info and jstat -gcutil <pid> 1000
- How to read it
GC.heap_infoprints heap total, used, region size and young regions.jstat -gcutilprints, once a second, old occupancy in percent (O), young collections (YGC), full collections (FGC) and concurrent cycles (CGC).FGCrising is the same finding asPause Fullin the log.
No GC log was configured and the problem is happening now
Check
jcmd <pid> VM.log what=gc*,safepoint output=file=/tmp/gc.log decorators=time,uptime,level,tags
- How to read it
- Turns unified logging on in the running JVM without a restart.
jcmd <pid> VM.log listshows the outputs now configured. The file starts at the moment of the command, so let it collect a few minutes of the problem.
Stops longer than any GC pause explains
Check
grep 'Safepoint' gc.log | awk -F'Total: ' '{print $2, $0}' | sort -n | tail -5- How to read it
- The five longest stops. If
Reaching safepointis a large part ofTotal, a thread was slow to stop. If the operation in quotes is not a collection, something else asked for the stop: a thread dump, a heap dump or a profiler.
A Spring Boot service with no shell access
Check
GET /actuator/metrics/jvm.gc.pause and /actuator/metrics/jvm.gc.live.data.size
- How to read it
jvm.gc.pauseis a timer with count, total time and max, tagged by cause and action.jvm.gc.live.data.sizeagainstjvm.gc.max.data.sizeis the live set as a share of the old generation's limit.
Key takeaways
- A collector in trouble is almost always a heap too small for its live set plus the room its allocation rate needs. Measure both from the log before changing anything.
- Read the kind and the cause on each line:
Pause YoungandMixedare routine,(Evacuation Failure)andPause Fullare sizing findings,G1 Humongous Allocationis a large-array finding. - The
aftervalue of a full collection is the live set. Allocation rate is the heap growth between two collections divided by the time between them. - Tune in order: heap size, live set, allocation rate, pause target. In the real experiment a 5 ms pause target left 14 full collections; a larger heap left none.
- Do not fix the young generation size with G1, and do not port flag lists from older JDKs.
- Log
safepointbesidegc*, with wall-clock time. The stop a request experiences is the safepoint total, not the GC duration. - On ZGC the equivalent of a full collection is an allocation stall. It is not a pause and pause metrics will not show it.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 4
A service with -Xmx4g on G1 logs, every few seconds: Pause Young (Mixed) (G1 Evacuation Pause) (Evacuation Failure) 4071M->4069M(4096M) 3.1ms followed by Pause Full (G1 Compaction Pause) 4069M->3480M(4096M) 2210.4ms. The team's p99 target is 300 ms.
What is the first change to make?
Scenario 2 of 4
Two consecutive lines of a healthy G1 log read [120.400s] GC(310) Pause Young (Normal) (G1 Evacuation Pause) 2900M->1100M(4096M) 18.2ms and [123.400s] GC(311) Pause Young (Normal) (G1 Evacuation Pause) 2900M->1104M(4096M) 17.9ms.
What is the allocation rate of this service?
Scenario 3 of 4
A trading gateway on generational ZGC with a 16 GB heap reports occasional 40 ms delays. The GC log shows Pause Mark Start, Pause Mark End and Pause Relocate Start all below 0.05 ms, collection lines such as Minor Collection (Allocation Stall) 15890M(97%)->15600M(95%), and Allocation Stall (order-worker-7) 38.512ms.
How should the delay be classified and addressed?
Scenario 4 of 4
Latency spikes of about 250 ms appear on a G1 service every 30 seconds. The longest pause in the GC log is 22 ms. With -Xlog:safepoint added, the lines at the spike times read: Safepoint "G1CollectForAllocation", ... Reaching safepoint: 231004211 ns, ... At safepoint: 19800321 ns, ... Total: 250812004 ns.
What does this show?
References
References: Java SE 21 GC Tuning Guide: Garbage-First Garbage Collector Tuning ↗ · Java SE 21 GC Tuning Guide: Garbage-First (G1) Garbage Collector ↗ · Java SE 21 GC Tuning Guide: The Z Garbage Collector ↗ · The java command, JDK 21: Enable Logging with the JVM Unified Logging Framework, GC options ↗ · The jcmd command, JDK 21 (GC.heap_info, VM.log) ↗ · Spring Boot 3.5 reference: Actuator metrics (JVM metrics) ↗ · Report a problem