Skip to content

Path 2 · lesson 4 of 4

Intermediate30 minJava 21 (OpenJDK 21.0.13) · Spring Boot 3.5 · async-profiler 4.x (documented usage; not installed here)

JFR, GC logs and async-profiler

Three tools and the question each one answers: Java Flight Recorder for what the JVM and its threads were doing, GC logs for pauses, async-profiler for where CPU, allocation and waiting time go.

What you will learn

  • Say what each of the three tools records, what it costs, and which question it answers
  • Start a flight recording at launch and on a running process, and get the data out with jcmd
  • Read a recording on the command line with jfr summary, jfr print --events and jfr view
  • Explain why a CPU profile and a lock profile of the same program point at different methods
  • Describe async-profiler's CPU, allocation, lock and wall-clock modes and read a flame graph
  • Pick the right tool from a symptom: high CPU, long pauses, blocked threads, slow but idle

Before you start

The concept

A thread dump and a heap dump are photographs: the state of threads or of memory at one instant. Most performance questions are about a period of time. Where did the CPU go over the last minute? How long did threads wait for that lock, in total? What stopped the application at 14:02:11? Three tools answer questions of that kind on JDK 21, and they overlap less than it first appears.

Java Flight Recorder (JFR)GC log (unified logging)async-profiler
What it isAn event recorder built into the JVMText lines written by the JVMA sampling profiler loaded into the JVM as a native agent
What it recordsTimestamped events: GC, lock waits, thread parks, I/O, allocation samples, method samples, exceptions, your own eventsEvery collection: kind, cause, heap before and after, duration. With safepoint, every stopStack traces sampled on CPU time, on allocation, on lock contention or on wall-clock time
InstalledPart of the JDKPart of the JDKA separate download
OutputA binary .jfr file, read with the jfr command or JDK Mission ControlPlain text you can grepAn HTML flame graph, text, or a .jfr file
Suited to run alwaysYes, with the default settingsYesOn demand, for a profiling session
Answers bestWhat was the JVM doing, and what were threads waiting for?Did a pause cause this, and why did the pause happen?Which code path uses the CPU, allocates the memory, or holds up the thread?
The three tools side by side. JFR and the GC log should already be running when a problem starts. async-profiler is what you attach when you need the full call tree.

Why it exists

During an incident you rarely get a second attempt. The process is about to be restarted, or the load that causes the problem is passing. If nothing was recording, the evidence is gone, and what is left is a guess. That is the case for a continuous flight recording and a GC log on every service: when something goes wrong, the last hours are already on disk.

The second reason is that the obvious measurement is often the wrong one. The lab program below runs 8 threads that count page views in one shared map, on 2 CPUs. A CPU profile of it and a lock profile of it give different answers, and only one of them explains why it is slow.

JFRWhere the CPU went (jfr view hot-methods /tmp/contention.jfr)

Real JVM output
                               Java Methods that Executes the Most

Method                                                                            Samples Percent
--------------------------------------------------------------------------------- ------- -------
LockContention.spin(int)                                                              405  89.01%
LockContention.recordGlobal(String)                                                    12   2.64%
java.lang.Integer.getChars(int, int, byte[])                                           11   2.42%
java.util.HashMap.merge(Object, Object, BiFunction)                                     7   1.54%
89% of the CPU samples are in spin, the small piece of work each update does. Read alone, this says: make spin faster. It is a true statement about CPU and it misses the problem. A CPU profile only sees threads while they run, and in this program most threads are not running most of the time.

JFRWhere the time went (jfr view contention-by-site /tmp/contention.jfr)

Real JVM output
                               Contention by Site

StackTrace                                                Count    Avg.    Max.
--------------------------------------------------------- ----- ------- -------
LockContention.recordGlobal(String)                         894 22.7 ms 58.7 ms
The same recording, a different view. In 5 seconds, threads waited 894 times to enter a monitor in recordGlobal, on average 22.7 ms each and up to 58.7 ms. That is about 20 seconds of waiting in a 5 second run: several threads are blocked at any moment. The work inside the lock takes microseconds; the wait for the lock takes milliseconds. This is the finding. Only waits of at least 10 ms were recorded (the threshold of the profile settings), so the true count is higher.

How it works inside

JFR: an event stream from inside the JVM

JFR is compiled into HotSpot. The JVM and the class library emit events at points of interest, each with a start time, usually a duration, the thread, and often a stack trace. Events are written to thread-local buffers and flushed to disk in chunks, which is why recording is cheap. Which events are on, and above which threshold they are kept, is decided by a settings file. Two ship with the JDK in lib/jfr.

Settingdefault.jfcprofile.jfc
Intended useContinuous recording in productionShort investigations
Method sampling (jdk.ExecutionSample)Every 20 msEvery 10 ms
Lock waits kept (jdk.JavaMonitorEnter, jdk.ThreadPark)20 ms or longer10 ms or longer
Allocation samples (jdk.ObjectAllocationSample)Up to 150 per secondUp to 300 per second
Allocation stack for old object samplesOffOn
Overhead, per the JDK 21 troubleshooting guideVery low, less than 1%Usually less than 2%; not recommended as always-on for allocation-intensive applications
Values read from the two files in OpenJDK 21.0.13. The thresholds explain what a recording can and cannot contain: a lock wait of 5 ms is in neither.

Starting a recording

At launch, -XX:StartFlightRecording starts a recording with the JVM. Its parameters are comma-separated: filename, settings, duration, delay, maxage, maxsize, dumponexit, name and path-to-gc-roots. With no duration the recording runs until the JVM stops, keeping the most recent data.

Program outputA recording started at launch (java -Xmx256m -XX:StartFlightRecording=filename=/tmp/gcw/gc.jfr,settings=profile GcPressure.java 215 8)

Real JVM output
[0.793s][info][jfr,startup] Started recording 1. No limit specified, using maxsize=250MB as default.
[0.793s][info][jfr,startup]
[0.793s][info][jfr,startup] Use jcmd 106846 JFR.dump name=1 to copy recording data to file.

live set target 215 MB, max heap 256 MB
requests 7,446 in 8 s, allocated 1,600 MB (200 MB/s)
slowest five requests (ms): [30, 31, 36, 39, 43]
The JVM confirms the recording, tells you it will keep at most 250 MB of data because no limit was given, and prints the exact jcmd command for getting data out while it runs. The recording is named 1 because no name was set. The file given by filename is written when the JVM exits.

On a process that is already running, jcmd does the same without a restart. JFR.start begins a recording, JFR.check lists recordings, JFR.dump writes what has been recorded so far to a file and lets the recording continue, and JFR.stop ends it.

Program outputRecording a running process for 10 seconds with jcmd

Real JVM output
$ jcmd 329836 JFR.start name=incident settings=profile duration=10s filename=/tmp/incident.jfr
329836:
Started recording 1. The result will be written to:

/tmp/incident.jfr

$ jcmd 329836 JFR.check
329836:
Recording 1: name=incident duration=10s (running)

$ jcmd 329836 JFR.dump name=incident filename=/tmp/incident-early.jfr
329836:
Dumped recording "incident", 266.9 kB written to:

/tmp/incident-early.jfr

$ jcmd 329836 JFR.check
329836:
No available recordings.

Use jcmd 329836 JFR.start to start a recording.
Four commands in order. JFR.start with a duration writes the file by itself when the time is up. JFR.check shows it running. JFR.dump four seconds in copies the data so far (267 kB) without stopping anything: use it when you want to look before the recording ends. The last JFR.check, after 10 seconds, shows the recording has finished and been removed. jcmd must run as the same user as the JVM; in a container, run it inside the container.

Reading a recording on the command line

The jfr command ships with the JDK and needs no GUI. Three subcommands cover most of an investigation. jfr summary <file> lists the event types in the file with their counts. jfr print --events <names> <file> prints individual events in full; names can be given without the jdk. prefix, and --stack-depth controls how many frames are shown (5 by default). jfr view <view> <file> prints aggregated tables such as hot-methods, contention-by-site, gc-pauses and allocation-by-class; jfr view with no arguments lists them all.

JFRWhat is in a recording (jfr summary /tmp/contention.jfr, first rows)

Real JVM output
 Version: 2.1
 Chunks: 1
 Start: 2026-11-15 21:26:34 (UTC)
 Duration: 6 s

 Event Type                              Count  Size (bytes)
=============================================================
 jdk.ObjectAllocationSample                995         14844
 jdk.JavaMonitorEnter                      894         21456
 jdk.ModuleExport                          507          4945
 jdk.BooleanFlag                           505         15365
 jdk.GCPhaseParallel                       504         13102
 jdk.ExecutionSample                       455          5002
Start every investigation here. The table is sorted by count, and the unusual entry stands out: 894 jdk.JavaMonitorEnter events in 6 seconds. A healthy service records few or none, because only waits above the threshold are kept. jdk.ExecutionSample (455) is the method sampler behind hot-methods. Rows such as jdk.BooleanFlag are the JVM's configuration, written once per recording.

JFROne event in full (jfr print --events jdk.JavaMonitorEnter /tmp/contention.jfr)

Real JVM output
jdk.JavaMonitorEnter {
  startTime = 21:26:35.120 (2026-11-15)
  duration = 12.9 ms
  monitorClass = java.util.HashMap (classLoader = bootstrap)
  previousOwner = "view-counter-1" (javaThreadId = 24)
  address = 0x7E86CC004D10
  eventThread = "view-counter-7" (javaThreadId = 30)
  stackTrace = [
    LockContention.recordGlobal(String) line: 24
    LockContention.lambda$main$1(CountDownLatch, long, int, String, LongAdder) line: 58
    java.lang.Thread.run() line: 1583
    ...
  ]
}
One wait, with everything needed to act on it. eventThread is who waited: view-counter-7, for 12.9 ms. monitorClass is what it waited for: a java.util.HashMap used as a lock. previousOwner is who held it. stackTrace is where: LockContention.recordGlobal, line 24, inside the synchronized block. The address is the same in every event of this recording, which says all threads queue for one single lock object.

GC in a recording, and why the log is still worth having

JFRPause statistics from a recording (jfr view gc-pauses /tmp/gcw/gc.jfr; G1, live set 215 MB, -Xmx256m)

Real JVM output
GC Pauses
---------

Total Pause Time: 603 ms

Number of Pauses: 188

Minimum Pause Time: 0.0506 ms

Median Pause Time: 1.27 ms

Average Pause Time: 3.21 ms

P90 Pause Time: 7.83 ms

P95 Pause Time: 18.9 ms

P99 Pause Time: 41.8 ms

P99.9% Pause Time: 41.1 ms

Maximum Pause Time: 41.1 ms
The summary you would otherwise compute from the log with awk: 188 pauses, 603 ms in total over a 9 second recording, a median of 1.3 ms and a tail of 19 ms at P95. A median far below the P95 means two populations of pauses: many short young collections and a few long ones. The long ones here are full collections. (The P99 above the maximum is how the tool printed it for this small sample; treat the top percentiles as approximately the maximum.)

JFRThe long pauses, by phase and as an event (jfr view gc-pause-phases; jfr print --events jdk.GarbageCollection)

Real JVM output
Type                   Name                                            Average         P95     Longest Count      Total
---------------------- ------------------------------------------- ----------- ----------- ----------- ----- ----------
GC Phase Pause         GC Pause                                        4.08 ms     23.9 ms     41.1 ms   130     531 ms
GC Phase Pause         Pause Remark                                    2.41 ms     12.0 ms     9.76 ms    29    69.8 ms
GC Phase Pause         Pause Cleanup                                 0.0842 ms    0.168 ms    0.152 ms    29    2.44 ms
GC Phase Pause Level 1 Phase 4: Compact heap                           11.7 ms     18.1 ms     18.1 ms    13     152 ms
GC Phase Pause Level 1 Phase 1: Mark live objects                      5.46 ms     11.5 ms     11.5 ms    13    71.0 ms
...
jdk.GarbageCollection {
  startTime = 21:38:48.817 (2026-11-15)
  duration = 35.1 ms
  gcId = 41
  name = "G1Full"
  cause = "G1 Compaction Pause"
  sumOfPauses = 35.1 ms
  longestPause = 35.1 ms
  eventThread = "VM Thread" (osThreadId = 107013)
}
Phase 4: Compact heap and Phase 1: Mark live objects each occur 13 times: those are the phases of a full collection, so there were 13 of them, costing 152 ms and 71 ms. The event below is one of them, with the same name and cause the GC log prints (G1 Compaction Pause). The recording gives the statistics in one command. The log gives the order of events and the heap before and after each one as text, which is what you need to see (Evacuation Failure) leading into a full collection.

Use the GC log when

  • You need the exact sequence around one pause: what came before, heap before and after, the cause in brackets
  • You want safepoint detail: time to reach the safepoint, non-GC stops
  • You want something any tool can read, including grep on a host with nothing installed
  • You are looking at ZGC allocation stalls, which are log lines

Use JFR when

  • You need pause statistics over a period without parsing text
  • You want to line pauses up with what threads were doing: lock waits, I/O, your own request events
  • You want to know what was being allocated and where (allocation-by-class, allocation-by-site)
  • You are not sure GC is the cause at all and want one file that covers the other suspects

async-profiler

async-profiler is an open-source sampling profiler for HotSpot JVMs. Its project page describes it as a low-overhead profiler that does not suffer from the safepoint bias problem, and that also monitors non-Java threads such as GC and JIT compiler threads and shows native and kernel frames in stack traces. It is not part of the JDK: you download it, and the asprof launcher attaches its agent to a running JVM. It can also be loaded at startup with -agentpath.

ModeOptionWhat a sample meansUse it for
CPU-e cpu (the default)A stack trace of a thread that was on a CPU. By default one sample per 10 ms of CPU timeHigh CPU. Shows Java, JVM, native and kernel frames in one tree
Allocation-e allocA stack trace of an allocation, weighted by bytes. The top frame is the class of the allocated objectA high allocation rate: which call paths create the garbage
Lock-e lockA stack trace of a contended lock acquisition, weighted by the nanoseconds it took to enter. The top frame is the class of the lockThreads blocked on monitors and locks: which lock, reached from where
Wall clock-e wallA stack trace of every thread at a fixed interval, whatever its state: running, sleeping or blockedA request that is slow while CPU is low: where the time passes, including waits on I/O
The four modes used most. Descriptions follow the project's Profiling Modes documentation.
async-profiler commands (from its documentation; not run here)Shell
# CPU profile of process 8983 for 30 seconds, written as an interactive flame graphasprof -d 30 -f /tmp/flamegraph.html 8983# Allocation profile: which call paths allocate the most bytesasprof -d 30 -e alloc -f /tmp/alloc.html 8983# Lock profile, one flame per threadasprof -d 30 -e lock -t -f /tmp/lock.html 8983# Wall-clock profile, per thread, sampling every 50 msasprof -e wall -t -i 50ms -d 30 -f /tmp/wall.html 8983# Start and stop by hand around the scenario you want to captureasprof start -e cpu -f /tmp/cpu.html 8983asprof stop 8983# Load the agent at JVM startup instead of attachingjava -agentpath:/path/to/libasyncProfiler.so=start,event=cpu,file=profile.html -jar app.jar

Why it is written this way

  • -d is the duration in seconds, -e the event, -f the output file, -t splits the profile by thread, -i sets the sampling interval. An output name ending in .html selects the flame graph format.
  • The documentation recommends starting the JVM with -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints for better accuracy, unless the agent is loaded at JVM startup.
  • On Linux, kernel stack traces from a non-root process need kernel.perf_event_paranoid=1 and kernel.kptr_restrict=0. By default a Docker container restricts the perf_event_open system call; the project's page on profiling in containers lists the ways around it.
  • A file name ending in .jfr writes the samples in JFR format, so the same viewers work for both tools.

ProfileWhat async-profiler's text output looks like for the lock-contention program

Illustrative, not from a run
--- Execution profile ---
Total samples:           1012

--- 9010000000 (89.03%) ns, 901 samples
  [ 0] LockContention.spin
  [ 1] LockContention.recordGlobal
  [ 2] LockContention.lambda$main$1
  [ 3] java.lang.Thread.run

--- 270000000 (2.67%) ns, 27 samples
  [ 0] java.util.HashMap.merge
  [ 1] LockContention.recordGlobal
  [ 2] LockContention.lambda$main$1
  [ 3] java.lang.Thread.run

          ns  percent  samples  top
  ----------  -------  -------  ---
  9010000000   89.03%      901  LockContention.spin
   270000000    2.67%       27  java.util.HashMap.merge
Written for this lesson in the layout of the sample output in async-profiler's Getting Started guide; it was not produced by a run, and the numbers are invented to match the proportions of the real JFR hot-methods table. Each block is one distinct call stack with the share of samples it received, leaf frame first. The table at the end ranks leaf methods. As a CPU profile it would tell the same partial story as hot-methods: you would need -e lock or -e wall to see the waiting.

Reading a flame graph

  • Each box is a method (a stack frame). A box sits on top of the method that called it, so the vertical axis is stack depth.
  • Width is the share of samples in which that frame was on the stack: CPU time, bytes allocated, or time waiting, depending on the mode. Wide means expensive.
  • The horizontal axis is not time. Stacks are merged and sorted by name, so left to right carries no order. The async-profiler guide states that a flame graph does not preserve when a stack was taken, only how often it was observed.
  • Look for wide plateaus at the top. A frame that is wide and has nothing above it is where the samples were actually taken. A wide frame with many narrow children is a method that fans out.
  • Know the mode before you read. The same service gives a CPU flame graph dominated by serialisation and a wall-clock flame graph dominated by a socket read. Neither is wrong; they answer different questions.

Symptom to tool

SymptomFirst lookThen
CPU near 100%, throughput flatjfr view hot-methods, or a CPU flame graphIf GC threads or allocation-by-site stand out, it is allocation pressure: go to the GC log
Latency spikes at intervalsGC log with safepointjfr view gc-pauses for the distribution
Throughput does not rise with more threadsjfr view contention-by-site, or a thread dumpA lock flame graph for the call paths
Requests slow, CPU lowThread dump for the states; a wall-clock profileJFR socket and file events, jdk.ThreadPark
Heap grows over daysGC log trend, class histogramsjdk.OldObjectSample with path-to-gc-roots, or a heap dump
A specific business operation is slow now and thenYour own JFR event for itLine its slow occurrences up with GC and lock events in the same file
Where to start. The first look is the cheap one that is usually already recorded.

Architecture

Drawing diagram…

Flowchart. From a slow or CPU-hungry service, check whether slow moments match GC pauses, in which case use the GC log and JFR pause views. Otherwise ask whether the CPU is busy. If yes, separate application code from GC threads and use a CPU or allocation profile. If no, a thread dump shows whether threads are blocked on a monitor, waiting, or in a socket read, each leading to a different tool.

flowchart TD
  S["Service is slow or using too much CPU"] --> Q{"Is the CPU busy"}
  Q -->|"Yes"| C{"Application threads or GC threads"}
  Q -->|"No"| W{"What are request threads doing in a thread dump"}
  C -->|"Application code"| C1["CPU profile, JFR hot-methods or async-profiler cpu"]
  C -->|"GC threads"| C2["GC log, then allocation profile"]
  W -->|"BLOCKED on a monitor"| L["JFR contention-by-site or async-profiler lock"]
  W -->|"WAITING or TIMED_WAITING"| P["JFR ThreadPark events, check pools and queues"]
  W -->|"RUNNABLE in socket read"| N["Wall-clock profile, look at the downstream call"]
  S --> G{"Do slow moments match pauses"}
  G -->|"Yes"| G1["GC log with safepoint, JFR gc-pauses"]
  G -->|"No"| Q
Choosing the tool from the symptom: first whether the CPU is busy, then what the threads are doing.
  • The first split is CPU busy or not. A CPU profile is the right tool only when the CPU is the scarce resource. When threads are waiting, a CPU profile shows whatever little is still running and hides the cause.
  • Busy CPU has two owners. If the busy threads are GC threads, the code to fix is the code that allocates, and the allocation profile finds it; optimising the hot methods will not help.
  • Idle CPU with slow requests is a waiting problem, and thread states name the kind of wait: BLOCKED is a monitor, WAITING is a lock, queue or future, RUNNABLE in a native socket read is a remote call.
  • The pause check runs beside all of this, because pauses hit every thread and can look like any of the other cases from the outside.

Step by step

Drawing diagram…

Sequence diagram. An engineer starts a flight recording on a running JVM with jcmd, dumps the data recorded so far to a file, and reads it with the jfr tool in three steps: summary to see which events are frequent, a view to aggregate lock contention by site, and print to see individual events with the lock class, owner and stack trace.

sequenceDiagram
  participant E as Engineer
  participant C as jcmd
  participant J as JVM with JFR
  participant F as Recording file
  participant T as jfr tool
  E->>C: JFR.start name=incident settings=profile duration=60s
  C->>J: Start recording
  J-->>E: Started recording 1
  Note over J: Events go to thread buffers, then to disk chunks
  E->>C: JFR.dump name=incident filename=early.jfr
  J->>F: Copy of the data so far
  E->>T: jfr summary early.jfr
  T-->>E: Event counts, JavaMonitorEnter stands out
  E->>T: jfr view contention-by-site early.jfr
  T-->>E: One site, hundreds of waits of 20 ms
  E->>T: jfr print --events JavaMonitorEnter early.jfr
  T-->>E: Lock class, owner thread and stack trace
From an alert to a named lock in six commands, without restarting the service.
  1. Nothing here restarts or pauses the service. JFR.start and JFR.dump are rated low impact in the jcmd reference.
  2. The order of reading is always wide to narrow: summary for which event types are present in unusual numbers, a view to aggregate them, print for single events with stack traces.
  3. If a continuous recording was already running, the first two steps disappear: JFR.dump alone gives you the period before the alert, which is the part you cannot get any other way.

In a real system

A product-page service counts views per URL in memory and flushes the counts every minute. After the instance type was changed from 2 to 8 CPUs, throughput did not improve, and with the request pool raised from 8 to 32 threads it got slightly worse. CPU sits at about half. Each tool gets a turn.

Thread dumpThread dump during the problem (jcmd 4603 Thread.print), two of the eight workers

Real JVM output
"view-counter-0" #23 [4626] prio=5 os_prio=0 cpu=830.58ms elapsed=3.65s tid=0x00007e8754631690 nid=4626 waiting for monitor entry  [0x00007e872dbfc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at LockContention.recordGlobal(LockContention.java:23)
	- waiting to lock <0x00000000f3500540> (a java.util.HashMap)
	at LockContention.lambda$main$1(LockContention.java:58)
	at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)

"view-counter-1" #24 [4627] prio=5 os_prio=0 cpu=798.40ms elapsed=3.65s tid=0x00007e8754632810 nid=4627 waiting for monitor entry  [0x00007e872dafc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at LockContention.recordGlobal(LockContention.java:23)
	- waiting to lock <0x00000000f3500540> (a java.util.HashMap)
Seven of the eight worker threads in this dump are BLOCKED (on object monitor), all waiting to lock <0x00000000f3500540>, the same HashMap. Two more details: cpu=830.58ms elapsed=3.65s says this thread used the CPU for less than a quarter of its life. A dump is one instant, though. It shows that threads queue for this lock right now, not how much time that costs in total. For that you need the recording.
  1. Thread dump. Seven of eight threads blocked on one monitor in recordGlobal. A strong hint, from a single instant.
  2. JFR. jfr summary shows 894 jdk.JavaMonitorEnter events; jfr view contention-by-site puts all of them at LockContention.recordGlobal(String) with an average wait of 22.7 ms. That quantifies it: threads spend far more time waiting for the lock than working.
  3. CPU profile. hot-methods shows 89% in spin, the work done while holding the lock. That is not the cause, but it is the second half of the answer: the lock is held for too long because that work is inside it.
  4. Fix. Count in a ConcurrentHashMap of LongAdder and move the work out of the critical section. The lab measured 563,433 updates per second before and 1,132,185 after, on the same 2 CPUs.
  5. Confirm. Record again. JavaMonitorEnter should be absent from the summary or near zero.

Program outputBefore and after (java LockContention.java global | striped, 8 threads, 2 CPUs, 5 s)

Real JVM output
pid 4547, mode global, 8 threads, 2 CPUs
global: 2,817,166 updates in 5 s = 563,433 updates/s
pid 4575, mode striped, 8 threads, 2 CPUs
striped: 5,660,929 updates in 5 s = 1,132,185 updates/s
Twice the throughput on the same hardware with the same amount of work per update. With one global lock, adding threads adds waiters. The Investigate case on pinned CPU with flat traffic starts from a neighbouring symptom and uses the same tools.

Java and Spring Boot

What a service should ship with: a continuous flight recording and a GC log, a short script for capturing evidence during an incident, and application events for the operations the business cares about, so that a slow checkout can be lined up with a GC pause or a lock wait in the same file. The flag line was started on OpenJDK 21.0.13 and the event class is the lab program JfrGcLogsAsyncProfiler.java, compiled and run here; its real output follows.

deploy/jvm.envShell
JAVA_TOOL_OPTIONS="-Xms2g -Xmx2g \  -XX:StartFlightRecording=name=continuous,maxage=6h,maxsize=500m,dumponexit=true,filename=/var/dumps/ \  -Xlog:gc*,safepoint:file=/var/log/shop/gc.log:time,uptime,level,tags:filecount=5,filesize=20m"

Why it is written this way

  • No settings=, so the recording uses default.jfc, the configuration meant for continuous use.
  • maxage=6h,maxsize=500m keeps a rolling window: the last six hours or 500 MB, whichever is reached first. Old chunks are deleted.
  • dumponexit=true writes the recording when the JVM shuts down. With a directory as filename, the file gets a generated name containing the process id and a timestamp (here it was hotspot-pid-679665-id-1-2026_11_15_22_12_26.jfr).
  • The GC log and the recording complement each other; keep both. Both write to disk continuously, so the paths need a volume with room.

Wrong, then right

The page-view counter from the scenario, as a Spring Boot service. It is called on every product page request from the request thread pool. The first version is correct, simple, and scales to exactly one thread.

Incorrect implementation

src/main/java/com/example/shop/analytics/PageViewCounter.javaJava
package com.example.shop.analytics;import java.util.HashMap;import java.util.Map;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.stereotype.Service;@Servicepublic class PageViewCounter {    private static final Logger log = LoggerFactory.getLogger(PageViewCounter.class);    private final Map<String, Long> views = new HashMap<>();    public void record(String url) {        synchronized (views) {            long count = views.merge(url, 1L, Long::sum);            // Formatting and logging while every other request thread waits for this lock.            log.debug("view {} of {}", count, url);        }    }    public Map<String, Long> snapshotAndReset() {        synchronized (views) {            Map<String, Long> copy = new HashMap<>(views);            views.clear();            return copy;        }    }}

Why it is written this way

  • One lock guards the whole map, so two requests for two different URLs still exclude each other.
  • The critical section contains more than the update: boxing a Long, and a logging call. Whatever runs inside the block is time during which no other thread can count anything.

Why it fails

  • Only one thread can be inside synchronized (views) at a time. With 8 request threads, one works and up to seven wait; the real thread dump showed exactly seven BLOCKED on the same HashMap.
  • A thread that fails to get the lock is descheduled and has to be woken later. That hand-over costs far more than the update itself, which is why the JFR events show waits of about 13 ms and an average of 22.7 ms for a critical section that runs in microseconds.
  • Adding CPUs or threads makes it worse. Throughput is capped by how fast the lock can be passed from thread to thread, and more waiters mean more hand-overs. The real run measured 563,433 updates per second with the global lock against 1,132,185 without it, on the same 2 CPUs.
  • In production this is a service whose throughput stays flat when it is given more hardware, CPU well below 100%, request latency that rises with concurrency, and a thread dump where most request threads show waiting to lock the same address. A CPU profile points at the code inside the lock and never at the lock.

Corrected implementation

src/main/java/com/example/shop/analytics/PageViewCounter.javaJava
package com.example.shop.analytics;import java.util.HashMap;import java.util.Map;import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.atomic.LongAdder;import org.springframework.stereotype.Service;@Servicepublic class PageViewCounter {    private final Map<String, LongAdder> views = new ConcurrentHashMap<>();    public void record(String url) {        // No shared lock: threads counting different URLs do not meet, and threads        // counting the same URL add to separate cells inside the LongAdder.        views.computeIfAbsent(url, key -> new LongAdder()).increment();    }    public Map<String, Long> snapshotAndReset() {        Map<String, Long> copy = new HashMap<>();        views.forEach((url, adder) -> {            long count = adder.sumThenReset();            if (count > 0) copy.put(url, count);        });        return copy;    }}

Why it is written this way

  • ConcurrentHashMap.computeIfAbsent locks at most one bin of the table, and only when it has to insert. Once the adder exists, the lookup takes no lock.
  • LongAdder spreads increments over several internal cells under contention, so even many threads counting the same hot URL do not queue.
  • The debug log line is gone from the hot path. If per-view logging is needed, it does not belong inside a lock, and probably not on the request thread.
  • Compiled on JDK 21 against a stand-in for the Spring annotation. The throughput figures come from the lab program LockContention.java, which has the same two variants.

Trade-offs

  • The snapshot is no longer a single consistent instant. sumThenReset runs per URL while other threads keep counting, so a view that arrives during the flush lands either in this snapshot or the next one. No view is counted twice, but the snapshot is not a point-in-time cut across all URLs.
  • LongAdder uses more memory than a boxed Long once it has been contended, and its sum() is not an atomic read. It is the right type for counters that are written often and read rarely, which is this case.
  • Entries are never removed, so the map grows with the number of distinct URLs seen. If URLs are unbounded (query strings, crawlers), this needs a bound, or it becomes the leak from the previous lesson.
  • The synchronized version was easy to reason about: one lock, one invariant. The concurrent version is correct for independent counters only. If two values had to change together, it would need a different design, not a finer lock.

Configuration

Recording options and their defaults on OpenJDK 21, from the java, jcmd and jfr command references and from what the JVM printed when run here. async-profiler defaults are from its documentation.

-XX:StartFlightRecording=settings=Default default.jfc (in JAVA_HOME/lib/jfr)
What it does
The event settings file: which events are on, their thresholds and sampling periods.
When to change it
Keep default for continuous recording. Use settings=profile for short investigations; it lowers thresholds and samples more often.
-XX:StartFlightRecording=maxage= / maxsize=Default Documented as unlimited (0). With neither set, JDK 21.0.13 printed "No limit specified, using maxsize=250MB as default"
What it does
How much recent data a recording without a duration keeps on disk.
When to change it
Set both explicitly for a continuous recording, for example maxage=6h,maxsize=500m, so the retention is a decision and not an accident.
-XX:StartFlightRecording=duration=Default 0 (not limited)
What it does
Stop the recording after this time and write the file.
When to change it
Use for one-off captures, also with jcmd <pid> JFR.start duration=60s. Leave unset for a continuous recording.
-XX:StartFlightRecording=dumponexit=Default false
What it does
Write the recording to a file when the JVM shuts down.
When to change it
Enable for continuous recordings so that a normal shutdown or an exit on OutOfMemoryError leaves a file behind. A hard kill leaves nothing.
-XX:StartFlightRecording=path-to-gc-roots=Default false
What it does
Collect the reference chain to the GC root for old object samples when the recording ends or is dumped.
When to change it
Only when hunting a leak: the java reference describes collecting it as time-consuming. It can also be passed to jcmd <pid> JFR.dump for one dump.
jfr print --stack-depthDefault 5
What it does
Number of stack frames printed per event.
When to change it
Raise it (for example to 20) when five frames end inside a framework and do not reach your code. The recording itself stores up to 64 frames by default.
-Xlog:gc*,safepointDefault Not enabled
What it does
GC and safepoint logging through unified logging.
When to change it
On for every service, to a rotated file with the time decoration. See the lesson on pause analysis.
asprof -i (interval)Default CPU profiling samples at 100 Hz, every 10 ms of CPU time
What it does
Sampling interval of async-profiler.
When to change it
The default suits most sessions. For wall-clock profiling of many threads, a longer interval such as -i 50ms keeps the overhead and the output size down.

Hands-on exercise

One lock, eight threads: which tool sees what

Run a contended counter, predict what a CPU profile, a thread dump and a lock profile would each report, then change the threads, the CPUs and the lock and check the predictions against the simulation and the real captured output.

Steps · 0 of 6 done

8 threads, 2 CPUs, one lock

Simulation · not a real JVM

Change the number of threads and CPUs, the kind of lock and the share of work done inside it. Watch time running against time blocked, and the throughput that results.

Take it further in the simulation: Lock contention: eight threads, the throughput of one

What can go wrong

Reading a CPU profile to explain waiting
Symptom
The profile's hottest method gets optimised and throughput does not change. CPU was below 100% all along.
Cause
A CPU profile samples threads that are running. Threads blocked on a lock, a pool or a socket contribute nothing to it.
Fix
Check CPU utilisation first. If it is not saturated, use jfr view contention-by-site, a thread dump, or a lock or wall-clock profile.
Starting to record after the problem has passed
Symptom
The incident lasted three minutes at night. The recording started the next morning shows a healthy service.
Cause
Nothing was recording when it happened. JFR cannot reconstruct the past.
Fix
Run a continuous recording with maxage and maxsize on every instance, and dump it when an alert fires.
Concluding there is no contention because no events were recorded
Symptom
jfr summary shows no jdk.JavaMonitorEnter, yet a thread dump shows threads BLOCKED.
Cause
The default settings keep only waits of 20 ms or more (10 ms with profile). Many short waits add up without any single one crossing the threshold. Waits on java.util.concurrent locks are jdk.ThreadPark events, not monitor events.
Fix
Record with settings=profile, look at jdk.ThreadPark as well, and use several thread dumps or a lock profile to see short, frequent waits.
Leaving profile settings on permanently
Symptom
A service runs a few percent slower since "temporary" profiling flags were added, and its recordings are several times larger.
Cause
profile.jfc samples more often, keeps shorter events and records allocation stack traces. The JDK troubleshooting guide advises against running it always on allocation-intensive applications.
Fix
Continuous recording with default settings; profile through jcmd JFR.start with a duration.
Reading the horizontal axis of a flame graph as time
Symptom
Someone reports that a method "runs first and takes the first third of the request".
Cause
Flame graph stacks are merged and sorted by name. Position from left to right has no meaning; only width does.
Fix
Read width as share of samples and height as stack depth. For ordering in time, use events with timestamps: JFR events or request traces.

How to troubleshoot

You do not know whether a flight recording is running on a service

Check

jcmd <pid> JFR.check
How to read it
Lists each recording with its name, duration and state, or prints No available recordings with the command to start one.
Throughput does not rise with more threads or CPUs

Check

jcmd <pid> JFR.dump filename=/tmp/now.jfr   then   jfr view contention-by-site /tmp/now.jfr
How to read it
One row per code location where threads waited for a monitor, with the count and the average and maximum wait. A single site with hundreds of waits is a lock everyone queues for. No rows with default settings only means no wait reached 20 ms.
CPU is high and you need the hot code without installing anything

Check

jfr view hot-methods /tmp/now.jfr   and   jfr view allocation-by-site /tmp/now.jfr
How to read it
hot-methods ranks methods by execution samples. allocation-by-site ranks methods by allocation pressure. If one site dominates allocation and GC threads are busy, the CPU is going to garbage collection on behalf of that code.
You need to know whether pauses explain a latency tail

Check

jfr view gc-pauses /tmp/now.jfr   and   grep -E 'Pause Full|Evacuation Failure|Allocation Stall' gc.log
How to read it
The view gives the distribution: compare P99 and maximum pause with the latency you are explaining. The grep finds the pauses that are findings in themselves. If neither shows anything near the size of the spikes, GC is not the cause.
One event type in the summary looks unusual and you need the details

Check

jfr print --events jdk.JavaMonitorEnter --stack-depth 20 /tmp/now.jfr | head -60
How to read it
Each event is printed with its fields and stack. For lock events read monitorClass, previousOwner and the top application frame. Replace the event name with any name from jfr summary.

Key takeaways

  • JFR records what the JVM and its threads did as timestamped events, cheaply enough to run all the time with the default settings. Run it continuously with maxage and maxsize.
  • Start at launch with -XX:StartFlightRecording, or on a live process with jcmd <pid> JFR.start; JFR.dump gets the data out without stopping anything.
  • Read a recording wide to narrow: jfr summary, then jfr view <view>, then jfr print --events <name>.
  • The GC log is still the best record of a pause: its kind, cause, heap before and after, and the safepoint around it. JFR adds statistics and the link to what threads were doing.
  • async-profiler is the tool for a full call tree as a flame graph, in CPU, allocation, lock or wall-clock mode, including native and kernel frames. It is a separate install.
  • A CPU profile explains busy CPUs. It cannot explain waiting: in the lab it showed 89% in spin while the real cost was 894 lock waits averaging 22.7 ms.
  • Decide from the symptom: busy CPU, pauses, blocked threads, or slow with idle CPU each point at a different tool.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 4

An order service was moved from 4 to 16 CPUs and its request pool from 50 to 200 threads. Throughput rose by 5%. CPU utilisation is 30%. A CPU flame graph shows JSON serialisation as the widest frame, with 40% of samples.

What should the team do next?

Scenario 2 of 4

At 03:10 a service had a two-minute latency spike and recovered. It runs with -XX:StartFlightRecording=name=continuous,maxage=6h,maxsize=500m and -Xlog:gc*,safepoint:file=gc.log:time,uptime,level,tags. It is now 08:30 and the on-call engineer wants to know what happened.

Which action recovers the evidence?

Scenario 3 of 4

A recording made with default settings shows no jdk.JavaMonitorEnter events. Three thread dumps taken a second apart each show 20 to 30 of 64 request threads BLOCKED in InventoryCache.get, a different set of threads each time.

How do these two observations fit together?

Scenario 4 of 4

A service's CPU is at 95% with flat traffic. jfr view hot-methods spreads samples thinly over hundreds of methods with none above 4%. jfr view allocation-by-site shows ReportMapper.toDto with 71% of allocation pressure, and the GC log shows a young collection every 80 ms.

Where is the CPU going, and what do you change?

References

References: The jfr command, JDK 21 (print, view, summary) ↗ · The jcmd command, JDK 21 (JFR.start, JFR.check, JFR.dump, JFR.stop, Thread.print) ↗ · The java command, JDK 21 (-XX:StartFlightRecording, -Xlog) ↗ · Java SE 21 Troubleshooting Guide: Troubleshoot Memory Leaks (JFR overhead, old object samples) ↗ · async-profiler: project page and README ↗ · async-profiler documentation: Getting Started ↗ · async-profiler documentation: Profiling Modes ↗ · async-profiler documentation: FlameGraph interpretation ↗ · Report a problem