Path 2 · lesson 4 of 4
JFR, GC logs and async-profiler
Three tools and the question each one answers: Java Flight Recorder for what the JVM and its threads were doing, GC logs for pauses, async-profiler for where CPU, allocation and waiting time go.
What you will learn
- Say what each of the three tools records, what it costs, and which question it answers
- Start a flight recording at launch and on a running process, and get the data out with jcmd
- Read a recording on the command line with jfr summary, jfr print --events and jfr view
- Explain why a CPU profile and a lock profile of the same program point at different methods
- Describe async-profiler's CPU, allocation, lock and wall-clock modes and read a flame graph
- Pick the right tool from a symptom: high CPU, long pauses, blocked threads, slow but idle
Before you start
The concept
A thread dump and a heap dump are photographs: the state of threads or of memory at one instant. Most performance questions are about a period of time. Where did the CPU go over the last minute? How long did threads wait for that lock, in total? What stopped the application at 14:02:11? Three tools answer questions of that kind on JDK 21, and they overlap less than it first appears.
| Java Flight Recorder (JFR) | GC log (unified logging) | async-profiler | |
|---|---|---|---|
| What it is | An event recorder built into the JVM | Text lines written by the JVM | A sampling profiler loaded into the JVM as a native agent |
| What it records | Timestamped events: GC, lock waits, thread parks, I/O, allocation samples, method samples, exceptions, your own events | Every collection: kind, cause, heap before and after, duration. With safepoint, every stop | Stack traces sampled on CPU time, on allocation, on lock contention or on wall-clock time |
| Installed | Part of the JDK | Part of the JDK | A separate download |
| Output | A binary .jfr file, read with the jfr command or JDK Mission Control | Plain text you can grep | An HTML flame graph, text, or a .jfr file |
| Suited to run always | Yes, with the default settings | Yes | On demand, for a profiling session |
| Answers best | What was the JVM doing, and what were threads waiting for? | Did a pause cause this, and why did the pause happen? | Which code path uses the CPU, allocates the memory, or holds up the thread? |
Why it exists
During an incident you rarely get a second attempt. The process is about to be restarted, or the load that causes the problem is passing. If nothing was recording, the evidence is gone, and what is left is a guess. That is the case for a continuous flight recording and a GC log on every service: when something goes wrong, the last hours are already on disk.
The second reason is that the obvious measurement is often the wrong one. The lab program below runs 8 threads that count page views in one shared map, on 2 CPUs. A CPU profile of it and a lock profile of it give different answers, and only one of them explains why it is slow.
JFRWhere the CPU went (jfr view hot-methods /tmp/contention.jfr)
Real JVM outputJava Methods that Executes the Most Method Samples Percent --------------------------------------------------------------------------------- ------- ------- LockContention.spin(int) 405 89.01% LockContention.recordGlobal(String) 12 2.64% java.lang.Integer.getChars(int, int, byte[]) 11 2.42% java.util.HashMap.merge(Object, Object, BiFunction) 7 1.54%
spin, the small piece of work each update does. Read alone, this says: make spin faster. It is a true statement about CPU and it misses the problem. A CPU profile only sees threads while they run, and in this program most threads are not running most of the time.JFRWhere the time went (jfr view contention-by-site /tmp/contention.jfr)
Real JVM outputContention by Site StackTrace Count Avg. Max. --------------------------------------------------------- ----- ------- ------- LockContention.recordGlobal(String) 894 22.7 ms 58.7 ms
recordGlobal, on average 22.7 ms each and up to 58.7 ms. That is about 20 seconds of waiting in a 5 second run: several threads are blocked at any moment. The work inside the lock takes microseconds; the wait for the lock takes milliseconds. This is the finding. Only waits of at least 10 ms were recorded (the threshold of the profile settings), so the true count is higher.How it works inside
JFR: an event stream from inside the JVM
JFR is compiled into HotSpot. The JVM and the class library emit events at points of interest, each with a start time, usually a duration, the thread, and often a stack trace. Events are written to thread-local buffers and flushed to disk in chunks, which is why recording is cheap. Which events are on, and above which threshold they are kept, is decided by a settings file. Two ship with the JDK in lib/jfr.
| Setting | default.jfc | profile.jfc |
|---|---|---|
| Intended use | Continuous recording in production | Short investigations |
Method sampling (jdk.ExecutionSample) | Every 20 ms | Every 10 ms |
Lock waits kept (jdk.JavaMonitorEnter, jdk.ThreadPark) | 20 ms or longer | 10 ms or longer |
Allocation samples (jdk.ObjectAllocationSample) | Up to 150 per second | Up to 300 per second |
| Allocation stack for old object samples | Off | On |
| Overhead, per the JDK 21 troubleshooting guide | Very low, less than 1% | Usually less than 2%; not recommended as always-on for allocation-intensive applications |
Starting a recording
At launch, -XX:StartFlightRecording starts a recording with the JVM. Its parameters are comma-separated: filename, settings, duration, delay, maxage, maxsize, dumponexit, name and path-to-gc-roots. With no duration the recording runs until the JVM stops, keeping the most recent data.
Program outputA recording started at launch (java -Xmx256m -XX:StartFlightRecording=filename=/tmp/gcw/gc.jfr,settings=profile GcPressure.java 215 8)
Real JVM output[0.793s][info][jfr,startup] Started recording 1. No limit specified, using maxsize=250MB as default. [0.793s][info][jfr,startup] [0.793s][info][jfr,startup] Use jcmd 106846 JFR.dump name=1 to copy recording data to file. live set target 215 MB, max heap 256 MB requests 7,446 in 8 s, allocated 1,600 MB (200 MB/s) slowest five requests (ms): [30, 31, 36, 39, 43]
jcmd command for getting data out while it runs. The recording is named 1 because no name was set. The file given by filename is written when the JVM exits.On a process that is already running, jcmd does the same without a restart. JFR.start begins a recording, JFR.check lists recordings, JFR.dump writes what has been recorded so far to a file and lets the recording continue, and JFR.stop ends it.
Program outputRecording a running process for 10 seconds with jcmd
Real JVM output$ jcmd 329836 JFR.start name=incident settings=profile duration=10s filename=/tmp/incident.jfr 329836: Started recording 1. The result will be written to: /tmp/incident.jfr $ jcmd 329836 JFR.check 329836: Recording 1: name=incident duration=10s (running) $ jcmd 329836 JFR.dump name=incident filename=/tmp/incident-early.jfr 329836: Dumped recording "incident", 266.9 kB written to: /tmp/incident-early.jfr $ jcmd 329836 JFR.check 329836: No available recordings. Use jcmd 329836 JFR.start to start a recording.
JFR.start with a duration writes the file by itself when the time is up. JFR.check shows it running. JFR.dump four seconds in copies the data so far (267 kB) without stopping anything: use it when you want to look before the recording ends. The last JFR.check, after 10 seconds, shows the recording has finished and been removed. jcmd must run as the same user as the JVM; in a container, run it inside the container.Reading a recording on the command line
The jfr command ships with the JDK and needs no GUI. Three subcommands cover most of an investigation. jfr summary <file> lists the event types in the file with their counts. jfr print --events <names> <file> prints individual events in full; names can be given without the jdk. prefix, and --stack-depth controls how many frames are shown (5 by default). jfr view <view> <file> prints aggregated tables such as hot-methods, contention-by-site, gc-pauses and allocation-by-class; jfr view with no arguments lists them all.
JFRWhat is in a recording (jfr summary /tmp/contention.jfr, first rows)
Real JVM outputVersion: 2.1 Chunks: 1 Start: 2026-11-15 21:26:34 (UTC) Duration: 6 s Event Type Count Size (bytes) ============================================================= jdk.ObjectAllocationSample 995 14844 jdk.JavaMonitorEnter 894 21456 jdk.ModuleExport 507 4945 jdk.BooleanFlag 505 15365 jdk.GCPhaseParallel 504 13102 jdk.ExecutionSample 455 5002
jdk.JavaMonitorEnter events in 6 seconds. A healthy service records few or none, because only waits above the threshold are kept. jdk.ExecutionSample (455) is the method sampler behind hot-methods. Rows such as jdk.BooleanFlag are the JVM's configuration, written once per recording.JFROne event in full (jfr print --events jdk.JavaMonitorEnter /tmp/contention.jfr)
Real JVM outputjdk.JavaMonitorEnter {
startTime = 21:26:35.120 (2026-11-15)
duration = 12.9 ms
monitorClass = java.util.HashMap (classLoader = bootstrap)
previousOwner = "view-counter-1" (javaThreadId = 24)
address = 0x7E86CC004D10
eventThread = "view-counter-7" (javaThreadId = 30)
stackTrace = [
LockContention.recordGlobal(String) line: 24
LockContention.lambda$main$1(CountDownLatch, long, int, String, LongAdder) line: 58
java.lang.Thread.run() line: 1583
...
]
}eventThread is who waited: view-counter-7, for 12.9 ms. monitorClass is what it waited for: a java.util.HashMap used as a lock. previousOwner is who held it. stackTrace is where: LockContention.recordGlobal, line 24, inside the synchronized block. The address is the same in every event of this recording, which says all threads queue for one single lock object.GC in a recording, and why the log is still worth having
JFRPause statistics from a recording (jfr view gc-pauses /tmp/gcw/gc.jfr; G1, live set 215 MB, -Xmx256m)
Real JVM outputGC Pauses --------- Total Pause Time: 603 ms Number of Pauses: 188 Minimum Pause Time: 0.0506 ms Median Pause Time: 1.27 ms Average Pause Time: 3.21 ms P90 Pause Time: 7.83 ms P95 Pause Time: 18.9 ms P99 Pause Time: 41.8 ms P99.9% Pause Time: 41.1 ms Maximum Pause Time: 41.1 ms
awk: 188 pauses, 603 ms in total over a 9 second recording, a median of 1.3 ms and a tail of 19 ms at P95. A median far below the P95 means two populations of pauses: many short young collections and a few long ones. The long ones here are full collections. (The P99 above the maximum is how the tool printed it for this small sample; treat the top percentiles as approximately the maximum.)JFRThe long pauses, by phase and as an event (jfr view gc-pause-phases; jfr print --events jdk.GarbageCollection)
Real JVM outputType Name Average P95 Longest Count Total
---------------------- ------------------------------------------- ----------- ----------- ----------- ----- ----------
GC Phase Pause GC Pause 4.08 ms 23.9 ms 41.1 ms 130 531 ms
GC Phase Pause Pause Remark 2.41 ms 12.0 ms 9.76 ms 29 69.8 ms
GC Phase Pause Pause Cleanup 0.0842 ms 0.168 ms 0.152 ms 29 2.44 ms
GC Phase Pause Level 1 Phase 4: Compact heap 11.7 ms 18.1 ms 18.1 ms 13 152 ms
GC Phase Pause Level 1 Phase 1: Mark live objects 5.46 ms 11.5 ms 11.5 ms 13 71.0 ms
...
jdk.GarbageCollection {
startTime = 21:38:48.817 (2026-11-15)
duration = 35.1 ms
gcId = 41
name = "G1Full"
cause = "G1 Compaction Pause"
sumOfPauses = 35.1 ms
longestPause = 35.1 ms
eventThread = "VM Thread" (osThreadId = 107013)
}Phase 4: Compact heap and Phase 1: Mark live objects each occur 13 times: those are the phases of a full collection, so there were 13 of them, costing 152 ms and 71 ms. The event below is one of them, with the same name and cause the GC log prints (G1 Compaction Pause). The recording gives the statistics in one command. The log gives the order of events and the heap before and after each one as text, which is what you need to see (Evacuation Failure) leading into a full collection.Use the GC log when
- You need the exact sequence around one pause: what came before, heap before and after, the cause in brackets
- You want safepoint detail: time to reach the safepoint, non-GC stops
- You want something any tool can read, including
grepon a host with nothing installed - You are looking at ZGC allocation stalls, which are log lines
Use JFR when
- You need pause statistics over a period without parsing text
- You want to line pauses up with what threads were doing: lock waits, I/O, your own request events
- You want to know what was being allocated and where (
allocation-by-class,allocation-by-site) - You are not sure GC is the cause at all and want one file that covers the other suspects
async-profiler
async-profiler is an open-source sampling profiler for HotSpot JVMs. Its project page describes it as a low-overhead profiler that does not suffer from the safepoint bias problem, and that also monitors non-Java threads such as GC and JIT compiler threads and shows native and kernel frames in stack traces. It is not part of the JDK: you download it, and the asprof launcher attaches its agent to a running JVM. It can also be loaded at startup with -agentpath.
| Mode | Option | What a sample means | Use it for |
|---|---|---|---|
| CPU | -e cpu (the default) | A stack trace of a thread that was on a CPU. By default one sample per 10 ms of CPU time | High CPU. Shows Java, JVM, native and kernel frames in one tree |
| Allocation | -e alloc | A stack trace of an allocation, weighted by bytes. The top frame is the class of the allocated object | A high allocation rate: which call paths create the garbage |
| Lock | -e lock | A stack trace of a contended lock acquisition, weighted by the nanoseconds it took to enter. The top frame is the class of the lock | Threads blocked on monitors and locks: which lock, reached from where |
| Wall clock | -e wall | A stack trace of every thread at a fixed interval, whatever its state: running, sleeping or blocked | A request that is slow while CPU is low: where the time passes, including waits on I/O |
# CPU profile of process 8983 for 30 seconds, written as an interactive flame graphasprof -d 30 -f /tmp/flamegraph.html 8983# Allocation profile: which call paths allocate the most bytesasprof -d 30 -e alloc -f /tmp/alloc.html 8983# Lock profile, one flame per threadasprof -d 30 -e lock -t -f /tmp/lock.html 8983# Wall-clock profile, per thread, sampling every 50 msasprof -e wall -t -i 50ms -d 30 -f /tmp/wall.html 8983# Start and stop by hand around the scenario you want to captureasprof start -e cpu -f /tmp/cpu.html 8983asprof stop 8983# Load the agent at JVM startup instead of attachingjava -agentpath:/path/to/libasyncProfiler.so=start,event=cpu,file=profile.html -jar app.jarWhy it is written this way
-dis the duration in seconds,-ethe event,-fthe output file,-tsplits the profile by thread,-isets the sampling interval. An output name ending in.htmlselects the flame graph format.- The documentation recommends starting the JVM with
-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepointsfor better accuracy, unless the agent is loaded at JVM startup. - On Linux, kernel stack traces from a non-root process need
kernel.perf_event_paranoid=1andkernel.kptr_restrict=0. By default a Docker container restricts theperf_event_opensystem call; the project's page on profiling in containers lists the ways around it. - A file name ending in
.jfrwrites the samples in JFR format, so the same viewers work for both tools.
ProfileWhat async-profiler's text output looks like for the lock-contention program
Illustrative, not from a run--- Execution profile ---
Total samples: 1012
--- 9010000000 (89.03%) ns, 901 samples
[ 0] LockContention.spin
[ 1] LockContention.recordGlobal
[ 2] LockContention.lambda$main$1
[ 3] java.lang.Thread.run
--- 270000000 (2.67%) ns, 27 samples
[ 0] java.util.HashMap.merge
[ 1] LockContention.recordGlobal
[ 2] LockContention.lambda$main$1
[ 3] java.lang.Thread.run
ns percent samples top
---------- ------- ------- ---
9010000000 89.03% 901 LockContention.spin
270000000 2.67% 27 java.util.HashMap.mergehot-methods: you would need -e lock or -e wall to see the waiting.Reading a flame graph
- Each box is a method (a stack frame). A box sits on top of the method that called it, so the vertical axis is stack depth.
- Width is the share of samples in which that frame was on the stack: CPU time, bytes allocated, or time waiting, depending on the mode. Wide means expensive.
- The horizontal axis is not time. Stacks are merged and sorted by name, so left to right carries no order. The async-profiler guide states that a flame graph does not preserve when a stack was taken, only how often it was observed.
- Look for wide plateaus at the top. A frame that is wide and has nothing above it is where the samples were actually taken. A wide frame with many narrow children is a method that fans out.
- Know the mode before you read. The same service gives a CPU flame graph dominated by serialisation and a wall-clock flame graph dominated by a socket read. Neither is wrong; they answer different questions.
Symptom to tool
| Symptom | First look | Then |
|---|---|---|
| CPU near 100%, throughput flat | jfr view hot-methods, or a CPU flame graph | If GC threads or allocation-by-site stand out, it is allocation pressure: go to the GC log |
| Latency spikes at intervals | GC log with safepoint | jfr view gc-pauses for the distribution |
| Throughput does not rise with more threads | jfr view contention-by-site, or a thread dump | A lock flame graph for the call paths |
| Requests slow, CPU low | Thread dump for the states; a wall-clock profile | JFR socket and file events, jdk.ThreadPark |
| Heap grows over days | GC log trend, class histograms | jdk.OldObjectSample with path-to-gc-roots, or a heap dump |
| A specific business operation is slow now and then | Your own JFR event for it | Line its slow occurrences up with GC and lock events in the same file |
Architecture
Drawing diagram…
Flowchart. From a slow or CPU-hungry service, check whether slow moments match GC pauses, in which case use the GC log and JFR pause views. Otherwise ask whether the CPU is busy. If yes, separate application code from GC threads and use a CPU or allocation profile. If no, a thread dump shows whether threads are blocked on a monitor, waiting, or in a socket read, each leading to a different tool.
flowchart TD
S["Service is slow or using too much CPU"] --> Q{"Is the CPU busy"}
Q -->|"Yes"| C{"Application threads or GC threads"}
Q -->|"No"| W{"What are request threads doing in a thread dump"}
C -->|"Application code"| C1["CPU profile, JFR hot-methods or async-profiler cpu"]
C -->|"GC threads"| C2["GC log, then allocation profile"]
W -->|"BLOCKED on a monitor"| L["JFR contention-by-site or async-profiler lock"]
W -->|"WAITING or TIMED_WAITING"| P["JFR ThreadPark events, check pools and queues"]
W -->|"RUNNABLE in socket read"| N["Wall-clock profile, look at the downstream call"]
S --> G{"Do slow moments match pauses"}
G -->|"Yes"| G1["GC log with safepoint, JFR gc-pauses"]
G -->|"No"| Q- The first split is CPU busy or not. A CPU profile is the right tool only when the CPU is the scarce resource. When threads are waiting, a CPU profile shows whatever little is still running and hides the cause.
- Busy CPU has two owners. If the busy threads are GC threads, the code to fix is the code that allocates, and the allocation profile finds it; optimising the hot methods will not help.
- Idle CPU with slow requests is a waiting problem, and thread states name the kind of wait:
BLOCKEDis a monitor,WAITINGis a lock, queue or future,RUNNABLEin a native socket read is a remote call. - The pause check runs beside all of this, because pauses hit every thread and can look like any of the other cases from the outside.
Step by step
Drawing diagram…
Sequence diagram. An engineer starts a flight recording on a running JVM with jcmd, dumps the data recorded so far to a file, and reads it with the jfr tool in three steps: summary to see which events are frequent, a view to aggregate lock contention by site, and print to see individual events with the lock class, owner and stack trace.
sequenceDiagram participant E as Engineer participant C as jcmd participant J as JVM with JFR participant F as Recording file participant T as jfr tool E->>C: JFR.start name=incident settings=profile duration=60s C->>J: Start recording J-->>E: Started recording 1 Note over J: Events go to thread buffers, then to disk chunks E->>C: JFR.dump name=incident filename=early.jfr J->>F: Copy of the data so far E->>T: jfr summary early.jfr T-->>E: Event counts, JavaMonitorEnter stands out E->>T: jfr view contention-by-site early.jfr T-->>E: One site, hundreds of waits of 20 ms E->>T: jfr print --events JavaMonitorEnter early.jfr T-->>E: Lock class, owner thread and stack trace
- Nothing here restarts or pauses the service.
JFR.startandJFR.dumpare rated low impact in thejcmdreference. - The order of reading is always wide to narrow:
summaryfor which event types are present in unusual numbers, aviewto aggregate them,printfor single events with stack traces. - If a continuous recording was already running, the first two steps disappear:
JFR.dumpalone gives you the period before the alert, which is the part you cannot get any other way.
In a real system
A product-page service counts views per URL in memory and flushes the counts every minute. After the instance type was changed from 2 to 8 CPUs, throughput did not improve, and with the request pool raised from 8 to 32 threads it got slightly worse. CPU sits at about half. Each tool gets a turn.
Thread dumpThread dump during the problem (jcmd 4603 Thread.print), two of the eight workers
Real JVM output"view-counter-0" #23 [4626] prio=5 os_prio=0 cpu=830.58ms elapsed=3.65s tid=0x00007e8754631690 nid=4626 waiting for monitor entry [0x00007e872dbfc000] java.lang.Thread.State: BLOCKED (on object monitor) at LockContention.recordGlobal(LockContention.java:23) - waiting to lock <0x00000000f3500540> (a java.util.HashMap) at LockContention.lambda$main$1(LockContention.java:58) at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source) at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596) at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583) "view-counter-1" #24 [4627] prio=5 os_prio=0 cpu=798.40ms elapsed=3.65s tid=0x00007e8754632810 nid=4627 waiting for monitor entry [0x00007e872dafc000] java.lang.Thread.State: BLOCKED (on object monitor) at LockContention.recordGlobal(LockContention.java:23) - waiting to lock <0x00000000f3500540> (a java.util.HashMap)
BLOCKED (on object monitor), all waiting to lock <0x00000000f3500540>, the same HashMap. Two more details: cpu=830.58ms elapsed=3.65s says this thread used the CPU for less than a quarter of its life. A dump is one instant, though. It shows that threads queue for this lock right now, not how much time that costs in total. For that you need the recording.- Thread dump. Seven of eight threads blocked on one monitor in
recordGlobal. A strong hint, from a single instant. - JFR.
jfr summaryshows 894jdk.JavaMonitorEnterevents;jfr view contention-by-siteputs all of them atLockContention.recordGlobal(String)with an average wait of 22.7 ms. That quantifies it: threads spend far more time waiting for the lock than working. - CPU profile.
hot-methodsshows 89% inspin, the work done while holding the lock. That is not the cause, but it is the second half of the answer: the lock is held for too long because that work is inside it. - Fix. Count in a
ConcurrentHashMapofLongAdderand move the work out of the critical section. The lab measured 563,433 updates per second before and 1,132,185 after, on the same 2 CPUs. - Confirm. Record again.
JavaMonitorEntershould be absent from the summary or near zero.
Program outputBefore and after (java LockContention.java global | striped, 8 threads, 2 CPUs, 5 s)
Real JVM outputpid 4547, mode global, 8 threads, 2 CPUs global: 2,817,166 updates in 5 s = 563,433 updates/s pid 4575, mode striped, 8 threads, 2 CPUs striped: 5,660,929 updates in 5 s = 1,132,185 updates/s
Java and Spring Boot
What a service should ship with: a continuous flight recording and a GC log, a short script for capturing evidence during an incident, and application events for the operations the business cares about, so that a slow checkout can be lined up with a GC pause or a lock wait in the same file. The flag line was started on OpenJDK 21.0.13 and the event class is the lab program JfrGcLogsAsyncProfiler.java, compiled and run here; its real output follows.
JAVA_TOOL_OPTIONS="-Xms2g -Xmx2g \ -XX:StartFlightRecording=name=continuous,maxage=6h,maxsize=500m,dumponexit=true,filename=/var/dumps/ \ -Xlog:gc*,safepoint:file=/var/log/shop/gc.log:time,uptime,level,tags:filecount=5,filesize=20m"Why it is written this way
- No
settings=, so the recording usesdefault.jfc, the configuration meant for continuous use. maxage=6h,maxsize=500mkeeps a rolling window: the last six hours or 500 MB, whichever is reached first. Old chunks are deleted.dumponexit=truewrites the recording when the JVM shuts down. With a directory asfilename, the file gets a generated name containing the process id and a timestamp (here it washotspot-pid-679665-id-1-2026_11_15_22_12_26.jfr).- The GC log and the recording complement each other; keep both. Both write to disk continuously, so the paths need a volume with room.
#!/usr/bin/env bash# Capture what is needed to diagnose a live problem, in the order of increasing cost.# Usage: incident-capture.sh <pid> <output-dir>set -euo pipefailpid="$1"; out="$2"; stamp="$(date +%Y%m%dT%H%M%S)"mkdir -p "$out"# 1. Three thread dumps, 5 seconds apart: is anything stuck in the same place each time?for i in 1 2 3; do jcmd "$pid" Thread.print > "$out/threads-$stamp-$i.txt" sleep 5done# 2. What the continuous recording already holds: the minutes BEFORE this script was started.jcmd "$pid" JFR.dump name=continuous filename="$out/before-$stamp.jfr"# 3. One minute of detailed recording of the problem as it is now.jcmd "$pid" JFR.start name=incident settings=profile duration=60s filename="$out/incident-$stamp.jfr"# 4. Heap layout, cheap. No histogram and no heap dump here: both pause the application.jcmd "$pid" GC.heap_info > "$out/heap-$stamp.txt"echo "Wait 60 s for incident-$stamp.jfr, then: jfr summary / jfr view contention-by-site / jfr view gc-pauses"Why it is written this way
- Thread dumps come first because they are instant and often enough. Three of them distinguish a thread that is stuck from one that happened to be passing through.
- Step 2 is the reason to run a continuous recording: it is the only source for what happened before anyone was looking.
- Step 3 uses
profilesettings for one minute only. Lower thresholds and denser sampling give more detail at more cost. - Every
jcmdsubcommand used here was run against OpenJDK 21.0.13 for this lesson. The script as a whole was syntax-checked withbash -n, not run against a production service.
package com.example.shop.checkout;import jdk.jfr.Category;import jdk.jfr.Event;import jdk.jfr.Label;import jdk.jfr.Name;/** One checkout, recorded in the flight recording next to the JVM's own events. */@Name("shop.Checkout")@Label("Checkout")@Category({"Shop", "Orders"})public class CheckoutEvent extends Event { @Label("Order Id") public long orderId; @Label("Items") public int items; @Label("Outcome") public String outcome;}Why it is written this way
- A JFR event is a class that extends
jdk.jfr.Event. Its fields become the event's columns. Start time, duration, thread and stack trace are added by JFR. @Namesets the name used withjfr print --eventsandjfr view. Without it the name is the fully qualified class name.- Keep fields to primitives and short strings. Never put personal data, tokens or card numbers in an event: recordings are files that get copied around.
package com.example.shop.checkout;import org.springframework.stereotype.Service;@Servicepublic class CheckoutService { private final PaymentGateway payments; public CheckoutService(PaymentGateway payments) { this.payments = payments; } public Receipt checkout(Order order) { CheckoutEvent event = new CheckoutEvent(); event.begin(); try { Receipt receipt = payments.charge(order); event.outcome = "PAID"; return receipt; } catch (RuntimeException e) { event.outcome = e.getClass().getSimpleName(); throw e; } finally { event.orderId = order.id(); event.items = order.itemCount(); event.commit(); } }}Why it is written this way
begin()takes the start time andcommit()ends the event and writes it. If no recording is running, or the event type is disabled,commit()does nothing, so the code can stay in permanently.- The
finallyblock makes sure failed checkouts are recorded too, with the exception type as the outcome. PaymentGateway,OrderandReceiptare assumed to exist. The event handling is the same as in the lab program, which was run; this Spring class was compiled against stand-in types only.
Wrong, then right
The page-view counter from the scenario, as a Spring Boot service. It is called on every product page request from the request thread pool. The first version is correct, simple, and scales to exactly one thread.
Incorrect implementation
package com.example.shop.analytics;import java.util.HashMap;import java.util.Map;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.stereotype.Service;@Servicepublic class PageViewCounter { private static final Logger log = LoggerFactory.getLogger(PageViewCounter.class); private final Map<String, Long> views = new HashMap<>(); public void record(String url) { synchronized (views) { long count = views.merge(url, 1L, Long::sum); // Formatting and logging while every other request thread waits for this lock. log.debug("view {} of {}", count, url); } } public Map<String, Long> snapshotAndReset() { synchronized (views) { Map<String, Long> copy = new HashMap<>(views); views.clear(); return copy; } }}Why it is written this way
- One lock guards the whole map, so two requests for two different URLs still exclude each other.
- The critical section contains more than the update: boxing a
Long, and a logging call. Whatever runs inside the block is time during which no other thread can count anything.
Why it fails
- Only one thread can be inside
synchronized (views)at a time. With 8 request threads, one works and up to seven wait; the real thread dump showed exactly sevenBLOCKEDon the sameHashMap. - A thread that fails to get the lock is descheduled and has to be woken later. That hand-over costs far more than the update itself, which is why the JFR events show waits of about 13 ms and an average of 22.7 ms for a critical section that runs in microseconds.
- Adding CPUs or threads makes it worse. Throughput is capped by how fast the lock can be passed from thread to thread, and more waiters mean more hand-overs. The real run measured 563,433 updates per second with the global lock against 1,132,185 without it, on the same 2 CPUs.
- In production this is a service whose throughput stays flat when it is given more hardware, CPU well below 100%, request latency that rises with concurrency, and a thread dump where most request threads show
waiting to lockthe same address. A CPU profile points at the code inside the lock and never at the lock.
Corrected implementation
package com.example.shop.analytics;import java.util.HashMap;import java.util.Map;import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.atomic.LongAdder;import org.springframework.stereotype.Service;@Servicepublic class PageViewCounter { private final Map<String, LongAdder> views = new ConcurrentHashMap<>(); public void record(String url) { // No shared lock: threads counting different URLs do not meet, and threads // counting the same URL add to separate cells inside the LongAdder. views.computeIfAbsent(url, key -> new LongAdder()).increment(); } public Map<String, Long> snapshotAndReset() { Map<String, Long> copy = new HashMap<>(); views.forEach((url, adder) -> { long count = adder.sumThenReset(); if (count > 0) copy.put(url, count); }); return copy; }}Why it is written this way
ConcurrentHashMap.computeIfAbsentlocks at most one bin of the table, and only when it has to insert. Once the adder exists, the lookup takes no lock.LongAdderspreads increments over several internal cells under contention, so even many threads counting the same hot URL do not queue.- The debug log line is gone from the hot path. If per-view logging is needed, it does not belong inside a lock, and probably not on the request thread.
- Compiled on JDK 21 against a stand-in for the Spring annotation. The throughput figures come from the lab program
LockContention.java, which has the same two variants.
Trade-offs
- The snapshot is no longer a single consistent instant.
sumThenResetruns per URL while other threads keep counting, so a view that arrives during the flush lands either in this snapshot or the next one. No view is counted twice, but the snapshot is not a point-in-time cut across all URLs. LongAdderuses more memory than a boxedLongonce it has been contended, and itssum()is not an atomic read. It is the right type for counters that are written often and read rarely, which is this case.- Entries are never removed, so the map grows with the number of distinct URLs seen. If URLs are unbounded (query strings, crawlers), this needs a bound, or it becomes the leak from the previous lesson.
- The synchronized version was easy to reason about: one lock, one invariant. The concurrent version is correct for independent counters only. If two values had to change together, it would need a different design, not a finer lock.
Configuration
Recording options and their defaults on OpenJDK 21, from the java, jcmd and jfr command references and from what the JVM printed when run here. async-profiler defaults are from its documentation.
-XX:StartFlightRecording=settings=Default default.jfc (in JAVA_HOME/lib/jfr)
- What it does
- The event settings file: which events are on, their thresholds and sampling periods.
- When to change it
- Keep
defaultfor continuous recording. Usesettings=profilefor short investigations; it lowers thresholds and samples more often.
-XX:StartFlightRecording=maxage= / maxsize=Default Documented as unlimited (0). With neither set, JDK 21.0.13 printed "No limit specified, using maxsize=250MB as default"
- What it does
- How much recent data a recording without a duration keeps on disk.
- When to change it
- Set both explicitly for a continuous recording, for example
maxage=6h,maxsize=500m, so the retention is a decision and not an accident.
-XX:StartFlightRecording=duration=Default 0 (not limited)
- What it does
- Stop the recording after this time and write the file.
- When to change it
- Use for one-off captures, also with
jcmd <pid> JFR.start duration=60s. Leave unset for a continuous recording.
-XX:StartFlightRecording=dumponexit=Default false
- What it does
- Write the recording to a file when the JVM shuts down.
- When to change it
- Enable for continuous recordings so that a normal shutdown or an exit on
OutOfMemoryErrorleaves a file behind. A hard kill leaves nothing.
-XX:StartFlightRecording=path-to-gc-roots=Default false
- What it does
- Collect the reference chain to the GC root for old object samples when the recording ends or is dumped.
- When to change it
- Only when hunting a leak: the
javareference describes collecting it as time-consuming. It can also be passed tojcmd <pid> JFR.dumpfor one dump.
jfr print --stack-depthDefault 5
- What it does
- Number of stack frames printed per event.
- When to change it
- Raise it (for example to 20) when five frames end inside a framework and do not reach your code. The recording itself stores up to 64 frames by default.
-Xlog:gc*,safepointDefault Not enabled
- What it does
- GC and safepoint logging through unified logging.
- When to change it
- On for every service, to a rotated file with the
timedecoration. See the lesson on pause analysis.
asprof -i (interval)Default CPU profiling samples at 100 Hz, every 10 ms of CPU time
- What it does
- Sampling interval of async-profiler.
- When to change it
- The default suits most sessions. For wall-clock profiling of many threads, a longer interval such as
-i 50mskeeps the overhead and the output size down.
Hands-on exercise
One lock, eight threads: which tool sees what
Run a contended counter, predict what a CPU profile, a thread dump and a lock profile would each report, then change the threads, the CPUs and the lock and check the predictions against the simulation and the real captured output.
Steps · 0 of 6 done
8 threads, 2 CPUs, one lock
Simulation · not a real JVMChange the number of threads and CPUs, the kind of lock and the share of work done inside it. Watch time running against time blocked, and the throughput that results.
What you should see
With the global lock, most threads are blocked at any moment and throughput is set by the lock, not by the CPUs: adding threads adds waiters, and adding CPUs changes little. A CPU profile would name the work done inside the lock (in the real run, spin with 89% of samples); a lock profile names the lock site (recordGlobal, 894 waits averaging 22.7 ms); a thread dump shows the queue (7 of 8 threads BLOCKED). Shrinking the time inside the lock helps in proportion. With the striped lock threads rarely meet, the CPUs become the limit, and throughput roughly doubles as in the real run: 563,433 against 1,132,185 updates per second.
What can go wrong
Reading a CPU profile to explain waiting
- Symptom
- The profile's hottest method gets optimised and throughput does not change. CPU was below 100% all along.
- Cause
- A CPU profile samples threads that are running. Threads blocked on a lock, a pool or a socket contribute nothing to it.
- Fix
- Check CPU utilisation first. If it is not saturated, use
jfr view contention-by-site, a thread dump, or a lock or wall-clock profile.
Starting to record after the problem has passed
- Symptom
- The incident lasted three minutes at night. The recording started the next morning shows a healthy service.
- Cause
- Nothing was recording when it happened. JFR cannot reconstruct the past.
- Fix
- Run a continuous recording with
maxageandmaxsizeon every instance, and dump it when an alert fires.
Concluding there is no contention because no events were recorded
- Symptom
jfr summaryshows nojdk.JavaMonitorEnter, yet a thread dump shows threadsBLOCKED.- Cause
- The default settings keep only waits of 20 ms or more (10 ms with
profile). Many short waits add up without any single one crossing the threshold. Waits onjava.util.concurrentlocks arejdk.ThreadParkevents, not monitor events. - Fix
- Record with
settings=profile, look atjdk.ThreadParkas well, and use several thread dumps or a lock profile to see short, frequent waits.
Leaving profile settings on permanently
- Symptom
- A service runs a few percent slower since "temporary" profiling flags were added, and its recordings are several times larger.
- Cause
profile.jfcsamples more often, keeps shorter events and records allocation stack traces. The JDK troubleshooting guide advises against running it always on allocation-intensive applications.- Fix
- Continuous recording with
defaultsettings;profilethroughjcmd JFR.startwith aduration.
Reading the horizontal axis of a flame graph as time
- Symptom
- Someone reports that a method "runs first and takes the first third of the request".
- Cause
- Flame graph stacks are merged and sorted by name. Position from left to right has no meaning; only width does.
- Fix
- Read width as share of samples and height as stack depth. For ordering in time, use events with timestamps: JFR events or request traces.
How to troubleshoot
You do not know whether a flight recording is running on a service
Check
jcmd <pid> JFR.check
- How to read it
- Lists each recording with its name, duration and state, or prints
No available recordingswith the command to start one.
Throughput does not rise with more threads or CPUs
Check
jcmd <pid> JFR.dump filename=/tmp/now.jfr then jfr view contention-by-site /tmp/now.jfr
- How to read it
- One row per code location where threads waited for a monitor, with the count and the average and maximum wait. A single site with hundreds of waits is a lock everyone queues for. No rows with default settings only means no wait reached 20 ms.
CPU is high and you need the hot code without installing anything
Check
jfr view hot-methods /tmp/now.jfr and jfr view allocation-by-site /tmp/now.jfr
- How to read it
hot-methodsranks methods by execution samples.allocation-by-siteranks methods by allocation pressure. If one site dominates allocation and GC threads are busy, the CPU is going to garbage collection on behalf of that code.
You need to know whether pauses explain a latency tail
Check
jfr view gc-pauses /tmp/now.jfr and grep -E 'Pause Full|Evacuation Failure|Allocation Stall' gc.log
- How to read it
- The view gives the distribution: compare P99 and maximum pause with the latency you are explaining. The
grepfinds the pauses that are findings in themselves. If neither shows anything near the size of the spikes, GC is not the cause.
One event type in the summary looks unusual and you need the details
Check
jfr print --events jdk.JavaMonitorEnter --stack-depth 20 /tmp/now.jfr | head -60
- How to read it
- Each event is printed with its fields and stack. For lock events read
monitorClass,previousOwnerand the top application frame. Replace the event name with any name fromjfr summary.
Key takeaways
- JFR records what the JVM and its threads did as timestamped events, cheaply enough to run all the time with the default settings. Run it continuously with
maxageandmaxsize. - Start at launch with
-XX:StartFlightRecording, or on a live process withjcmd <pid> JFR.start;JFR.dumpgets the data out without stopping anything. - Read a recording wide to narrow:
jfr summary, thenjfr view <view>, thenjfr print --events <name>. - The GC log is still the best record of a pause: its kind, cause, heap before and after, and the safepoint around it. JFR adds statistics and the link to what threads were doing.
- async-profiler is the tool for a full call tree as a flame graph, in CPU, allocation, lock or wall-clock mode, including native and kernel frames. It is a separate install.
- A CPU profile explains busy CPUs. It cannot explain waiting: in the lab it showed 89% in
spinwhile the real cost was 894 lock waits averaging 22.7 ms. - Decide from the symptom: busy CPU, pauses, blocked threads, or slow with idle CPU each point at a different tool.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 4
An order service was moved from 4 to 16 CPUs and its request pool from 50 to 200 threads. Throughput rose by 5%. CPU utilisation is 30%. A CPU flame graph shows JSON serialisation as the widest frame, with 40% of samples.
What should the team do next?
Scenario 2 of 4
At 03:10 a service had a two-minute latency spike and recovered. It runs with -XX:StartFlightRecording=name=continuous,maxage=6h,maxsize=500m and -Xlog:gc*,safepoint:file=gc.log:time,uptime,level,tags. It is now 08:30 and the on-call engineer wants to know what happened.
Which action recovers the evidence?
Scenario 3 of 4
A recording made with default settings shows no jdk.JavaMonitorEnter events. Three thread dumps taken a second apart each show 20 to 30 of 64 request threads BLOCKED in InventoryCache.get, a different set of threads each time.
How do these two observations fit together?
Scenario 4 of 4
A service's CPU is at 95% with flat traffic. jfr view hot-methods spreads samples thinly over hundreds of methods with none above 4%. jfr view allocation-by-site shows ReportMapper.toDto with 71% of allocation pressure, and the GC log shows a young collection every 80 ms.
Where is the CPU going, and what do you change?
References
References: The jfr command, JDK 21 (print, view, summary) ↗ · The jcmd command, JDK 21 (JFR.start, JFR.check, JFR.dump, JFR.stop, Thread.print) ↗ · The java command, JDK 21 (-XX:StartFlightRecording, -Xlog) ↗ · Java SE 21 Troubleshooting Guide: Troubleshoot Memory Leaks (JFR overhead, old object samples) ↗ · async-profiler: project page and README ↗ · async-profiler documentation: Getting Started ↗ · async-profiler documentation: Profiling Modes ↗ · async-profiler documentation: FlameGraph interpretation ↗ · Report a problem