Lock contention: eight threads, the throughput of one
Eight threads share one synchronized map on two CPUs. Throughput stays at the level of one thread and seven threads sit BLOCKED. Find it in a thread dump and in JFR, then remove the shared lock.
The situation
A Spring Boot 3 service counts page views per route in a bean that every request passes through. The pod has 2 CPUs and the load test uses 8 worker threads. Throughput stops at about 563,000 updates per second. Raising the worker count from 8 to 16 changes nothing, and the CPU graph shows about 90% busy, so the team asks for a bigger pod. A thread dump taken during the test shows 7 of the 8 workers in state BLOCKED, all waiting for the same java.util.HashMap.
Read first: Deadlocks, livelocks, starvation and lock contention, Concurrent collections and ConcurrentHashMap, JFR, GC logs and async-profiler
Break it in the simulation
One lock, many threads
Simulation · not a real JVMStart with 8 threads on 2 CPUs and one global lock. Read the verdict, then move Threads and watch the chart stay flat. Switch Lock to striped to see the CPUs become the limit.
Guided run · 0 of 6 stages done
- Set Threads to 1. Leave CPUs at 2, Lock on global and Share of time inside the lock at 70%.
Look for
- Throughput is about 566k/s: one thread uses one CPU.
- Threads at a typical instant shows one box, RUN · lock. Nobody is BLOCKED.
- CPU utilisation is 50%: the second CPU has nothing to do.
- Move Threads to 8, the lab setting.
- Then try 16 and 32.
Look for
- Throughput stays at about 563k/s. It equals Lock limit, which is half of CPU limit (1,132k/s).
- The chart Throughput against thread count is flat from 2 threads on.
- The verdict names the lock as the bottleneck and says more threads will not help.
- Set Threads back to 8.
- Read the thread boxes, then the simulated Thread dump and the JFR: contention by site line.
Look for
- 7 BLOCKED, 1 RUNNABLE: the same count as the real dump from the lab.
- In the dump, seven threads say
waiting to lock <0x00000000f3500540> (a java.util.HashMap)and one sayslockedwith the same address. The address tells you it is one lock. - CPU utilisation is about 92% but only half of it is the work of the operations. A busy CPU graph does not prove the CPUs are the limit.
- Raise CPUs from 2 to 8.
- Put CPUs back to 2 and lower Share of time inside the lock from 70% to 30%.
Look for
- More CPUs: Throughput does not move. The lock limit does not depend on the CPU count.
- A shorter critical section: Lock limit rises above CPU limit and the verdict changes to the CPUs. Doing less inside the lock is a real fix.
- Set Share of time inside the lock back to 70% and Threads to 8.
- Switch Lock to striped.
Look for
- Throughput doubles to about 1,132k/s, the CPU limit.
- 0 BLOCKED. 2 threads are on a CPU and 6 are RUNNABLE but waiting for a CPU.
- The dump has no
waiting to lockline and the JFR line reports no contention site.
- With Lock on striped, raise CPUs step by step to 8.
- Then lower Threads to 4.
Look for
- Throughput now follows the CPUs: 8 CPUs give about 4,529k/s.
- With 4 threads on 8 CPUs the verdict names the thread count: now more threads do help.
Wrong, then right
Incorrect implementation
import java.util.HashMap;import java.util.Locale;import java.util.Map;import org.springframework.stereotype.Component;/** Counts page views per route. Called from a HandlerInterceptor on every request. */@Componentpublic class PageViewStats { private final Map<String, Long> views = new HashMap<>(); public synchronized void record(String path) { // Everything below runs while holding the one lock of this bean. String route = path.toLowerCase(Locale.ROOT).replaceAll("/\\d+", "/{id}"); views.merge(route, 1L, Long::sum); } public synchronized Map<String, Long> snapshot() { return new HashMap<>(views); }}Why it is written this way
synchronizedon an instance method locks the bean itself. The bean is a singleton, so every request thread in the service queues for one monitor.- The route is normalised with a regular expression while the lock is held. That work touches no shared state, yet it makes the critical section long.
- It is correct: no update is lost. That is why it passes every test and only shows up under load.
- Compiled with javac 21 against a stub of
@Component; not run inside Spring.
Why it fails
- A monitor admits one thread at a time. If an update keeps the lock busy for 1.77 µs, the bean can do at most 1 ÷ 1.77 µs = about 563,000 updates per second, on 2 CPUs or on 64.
- Threads that arrive while the lock is taken go to state BLOCKED. With 8 threads, 7 are waiting at any instant, which is what the dump shows. A ninth thread only makes the queue longer.
- The CPUs still look busy. In the lab the 8 workers used about 1.8 of 2 CPUs between them (the
cpu=andelapsed=fields of the dump), while the completed work was what one CPU can do. The rest goes into trying for the lock and switching threads. - Because CPU is high and throughput is flat, the usual reaction is to add CPUs or threads. Neither moves a limit set by one lock.
Corrected implementation
import java.util.Locale;import java.util.Map;import java.util.TreeMap;import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.atomic.LongAdder;import org.springframework.stereotype.Component;/** Counts page views per route. Called from a HandlerInterceptor on every request. */@Componentpublic class PageViewStats { private final ConcurrentHashMap<String, LongAdder> views = new ConcurrentHashMap<>(); public void record(String path) { // The expensive part needs no shared state, so it runs with no lock held. String route = path.toLowerCase(Locale.ROOT).replaceAll("/\\d+", "/{id}"); views.computeIfAbsent(route, k -> new LongAdder()).increment(); } /** Not one instant in time: counters keep moving while this runs. */ public Map<String, Long> snapshot() { Map<String, Long> copy = new TreeMap<>(); views.forEach((route, count) -> copy.put(route, count.sum())); return copy; }}Why it is written this way
- The normalisation runs before any shared structure is touched, so nothing is locked while it runs.
computeIfAbsent(...).increment()is the frequency-map idiom from the ConcurrentHashMap documentation. The mapping function only creates a LongAdder, so it is short.- LongAdder is built for many threads updating one sum. Its documentation says throughput under high contention is significantly higher than AtomicLong, at the cost of more memory.
- Compiled with javac 21 against a stub of
@Component; not run inside Spring.
Trade-offs
snapshot()is no longer one instant in time.LongAdder.sum()is documented as not being an atomic snapshot, and the map can change while it is copied. For statistics that is fine; for money it is not.- A LongAdder uses more memory than a Long, and the map still grows with the number of distinct routes. Normalise the keys or cap the map.
- You cannot update two counters as one atomic step any more. If an invariant spans several keys you need a lock again, held as briefly as possible.
- The fix moves the limit to the CPUs. At that point throughput is 1,132k/s on 2 CPUs and more threads still do not help; more CPUs or cheaper operations do.
Run it on a real JVM
LockContention.java starts 8 threads that count page views for 5 seconds. With global every update goes through one synchronized HashMap; with striped it uses a ConcurrentHashMap of LongAdder. The same work is done in both, so the difference in updates per second is the cost of the shared lock.
With Docker (CPU and memory limits)
cd labs/jvm/lock-contentiondocker build -t jvm-lab-lock-contention .# broken: one synchronized HashMapdocker run --rm --cpus=2 --memory=512m jvm-lab-lock-contention LockContention global# fixed: ConcurrentHashMap + LongAdderdocker run --rm --cpus=2 --memory=512m jvm-lab-lock-contention LockContention striped# a 60 second run with a JFR recording, to inspect from a second terminaldocker run --rm --name lock-lab --cpus=2 --memory=512m jvm-lab-lock-contention -XX:StartFlightRecording=filename=/tmp/contention.jfr,settings=profile LockContention global 8 60# second terminal: thread dump, then a copy of the recording and its contention viewdocker exec lock-lab jcmd 1 Thread.printdocker exec lock-lab jcmd 1 JFR.dump filename=/tmp/dump.jfrdocker exec lock-lab jfr view contention-by-site /tmp/dump.jfrdocker exec lock-lab jfr print --events jdk.JavaMonitorEnter /tmp/dump.jfrWithout Docker, on a local JDK 21
java -XX:ActiveProcessorCount=2 LockContention.java globaljava -XX:ActiveProcessorCount=2 LockContention.java stripedjava -XX:ActiveProcessorCount=2 -XX:StartFlightRecording=filename=/tmp/contention.jfr,settings=profile LockContention.java global 8 60# second terminal; the program prints its pid on the first linejcmd <pid> Thread.printjcmd <pid> JFR.dump filename=/tmp/dump.jfrjfr view contention-by-site /tmp/dump.jfrjfr print --events jdk.JavaMonitorEnter /tmp/dump.jfrThe program: LockContention.java
import java.util.HashMap;import java.util.Map;import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.CountDownLatch;import java.util.concurrent.atomic.LongAdder;/** * Lab 6: CPU saturation and lock contention. * * Worker threads count page views per URL for a fixed time. * * java LockContention.java global -> one synchronized HashMap: every thread queues for the same lock * java LockContention.java striped -> ConcurrentHashMap + LongAdder: threads rarely meet * * Record it: java -XX:StartFlightRecording=filename=/tmp/contention.jfr,settings=profile LockContention.java global * Read it: jfr print --events jdk.JavaMonitorEnter /tmp/contention.jfr | head -40 */public class LockContention { static final Map<String, Long> globalMap = new HashMap<>(); static final Map<String, LongAdder> striped = new ConcurrentHashMap<>(); static void recordGlobal(String url) { synchronized (globalMap) { globalMap.merge(url, 1L, Long::sum); spin(2_000); // a little work done while holding the lock, e.g. formatting a log line } } static void recordStriped(String url) { striped.computeIfAbsent(url, k -> new LongAdder()).increment(); spin(2_000); // the same work, outside any shared lock } static long sink; static void spin(int n) { long x = 0; for (int i = 0; i < n; i++) x += i ^ (x >>> 3); sink = x; } public static void main(String[] args) throws Exception { String mode = args.length > 0 ? args[0] : "global"; int threads = args.length > 1 ? Integer.parseInt(args[1]) : 8; int seconds = args.length > 2 ? Integer.parseInt(args[2]) : 5; System.out.println("pid " + ProcessHandle.current().pid() + ", mode " + mode + ", " + threads + " threads, " + Runtime.getRuntime().availableProcessors() + " CPUs"); LongAdder done = new LongAdder(); CountDownLatch start = new CountDownLatch(1); long end = System.nanoTime() + seconds * 1_000_000_000L; Thread[] workers = new Thread[threads]; for (int t = 0; t < threads; t++) { int id = t; workers[t] = new Thread(() -> { try { start.await(); } catch (InterruptedException e) { return; } int i = 0; while (System.nanoTime() < end) { String url = "/product/" + ((id * 31 + i++) % 200); if (mode.equals("global")) recordGlobal(url); else recordStriped(url); done.increment(); } }, "view-counter-" + t); workers[t].start(); } start.countDown(); for (Thread w : workers) w.join(); System.out.printf("%s: %,d updates in %d s = %,d updates/s%n", mode, done.sum(), seconds, done.sum() / seconds); }}The Dockerfile
# Java 21 lab image. Build: docker build -t jvm-lab-lock-contention .FROM eclipse-temurin:21-jdkWORKDIR /labCOPY LockContention.java .RUN javac -d out LockContention.java# The JVM reads the container's CPU and memory limits (container support is on by default),# so --cpus and --memory on "docker run" change what Runtime.availableProcessors() and the default heap size report.ENTRYPOINT ["java", "-cp", "out"]CMD ["LockContention"]$ java -XX:ActiveProcessorCount=2 LockContention.java global
Program outputOne synchronized HashMap
Real JVM outputpid 4547, mode global, 8 threads, 2 CPUs global: 2,817,166 updates in 5 s = 563,433 updates/s
$ java -XX:ActiveProcessorCount=2 LockContention.java striped
Program outputConcurrentHashMap + LongAdder
Real JVM outputpid 4575, mode striped, 8 threads, 2 CPUs striped: 5,660,929 updates in 5 s = 1,132,185 updates/s
$ jcmd <pid> Thread.print
Thread dumpThread dump during a global run
Real JVM output"view-counter-0" #23 [4626] prio=5 os_prio=0 cpu=830.58ms elapsed=3.65s tid=0x00007e8754631690 nid=4626 waiting for monitor entry [0x00007e872dbfc000] java.lang.Thread.State: BLOCKED (on object monitor) at LockContention.recordGlobal(LockContention.java:23) - waiting to lock <0x00000000f3500540> (a java.util.HashMap) at LockContention.lambda$main$1(LockContention.java:58) at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source) at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596) at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583) "view-counter-1" #24 [4627] prio=5 os_prio=0 cpu=798.40ms elapsed=3.65s tid=0x00007e8754632810 nid=4627 waiting for monitor entry [0x00007e872dafc000] java.lang.Thread.State: BLOCKED (on object monitor) at LockContention.recordGlobal(LockContention.java:23) - waiting to lock <0x00000000f3500540> (a java.util.HashMap) at LockContention.lambda$main$1(LockContention.java:58) at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source) at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596) at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583) "view-counter-2" #25 [4628] prio=5 os_prio=0 cpu=824.97ms elapsed=3.65s tid=0x00007e87546339d0 nid=4628 waiting for monitor entry [0x00007e872d9fc000] java.lang.Thread.State: BLOCKED (on object monitor) at LockContention.recordGlobal(LockContention.java:23) - waiting to lock <0x00000000f3500540> (a java.util.HashMap) at LockContention.lambda$main$1(LockContention.java:58) at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source) at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596) at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583) "view-counter-3" #26 [4629] prio=5 os_prio=0 cpu=762.81ms elapsed=3.65s tid=0x00007e8754634b50 nid=4629 waiting for monitor entry [0x00007e872d8fc000] java.lang.Thread.State: BLOCKED (on object monitor) at LockContention.recordGlobal(LockContention.java:23) - waiting to lock <0x00000000f3500540> (a java.util.HashMap) at LockContention.lambda$main$1(LockContention.java:58) at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source) at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596) at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583) "view-counter-4" #27 [4630] prio=5 os_prio=0 cpu=834.12ms elapsed=3.65s tid=0x00007e8754635d60 nid=4630 waiting for monitor entry [0x00007e872d7fc000] java.lang.Thread.State: BLOCKED (on object monitor) at LockContention.recordGlobal(LockContention.java:23) - waiting to lock <0x00000000f3500540> (a java.util.HashMap) at LockContention.lambda$main$1(LockContention.java:58) at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source) at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596) at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583) "view-counter-5" #28 [4631] prio=5 os_prio=0 cpu=890.45ms elapsed=3.65s tid=0x00007e87546375f0 nid=4631 runnable [0x00007e872d6fb000] java.lang.Thread.State: RUNNABLE at LockContention.spin(LockContention.java:37) at LockContention.recordGlobal(LockContention.java:25) - locked <0x00000000f3500540> (a java.util.HashMap) at LockContention.lambda$main$1(LockContention.java:58) at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source) at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596) at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)
$ jfr view contention-by-site /tmp/contention.jfr
JFRJFR: contention by site
Real JVM output
Contention by Site
StackTrace Count Avg. Max.
--------------------------------------------------------- ----- ------- -------
LockContention.recordGlobal(String) 894 22.7 ms 58.7 ms$ jfr print --events jdk.JavaMonitorEnter /tmp/contention.jfr
JFRJFR: one jdk.JavaMonitorEnter event
Real JVM outputjdk.JavaMonitorEnter {
startTime = 21:26:35.120 (2026-11-15)
duration = 12.9 ms
monitorClass = java.util.HashMap (classLoader = bootstrap)
previousOwner = "view-counter-1" (javaThreadId = 24)
address = 0x7E86CC004D10
eventThread = "view-counter-7" (javaThreadId = 30)
stackTrace = [
LockContention.recordGlobal(String) line: 24
LockContention.lambda$main$1(CountDownLatch, long, int, String, LongAdder) line: 58
java.lang.Thread.run() line: 1583
...
]
}$ jfr view hot-methods /tmp/contention.jfr
ProfileJFR: hot methods
Real JVM output
Java Methods that Executes the Most
Method Samples Percent
--------------------------------------------------------------------------------- ------- -------
LockContention.spin(int) 405 89.01%
LockContention.recordGlobal(String) 12 2.64%
java.lang.Integer.getChars(int, int, byte[]) 11 2.42%
java.util.HashMap.merge(Object, Object, BiFunction) 7 1.54%
LockContention.lambda$main$1(CountDownLatch, long, int, String, LongAdder) 6 1.32%Real run and simulation: what each one shows
- The captured files come from OpenJDK 21.0.13 run directly on Linux x86-64 with -XX:ActiveProcessorCount=2, not in Docker. The Docker commands above were not executed when this was written.
- The real run shows what the model cannot: waits of very different lengths (an average of 22.7 ms and a maximum of 58.7 ms in JFR), because a monitor is not fair and a thread can be passed over many times. The widget only knows averages.
- In the lab program almost the whole update happens inside the lock. The widget starts at a 70% share and adds a queueing cost to reach the same 563k/s, so its split between the two is an assumption, not a measurement.
- The simulation shows what one run cannot: the whole curve from 1 to 32 threads, and what happens when you change the CPU count or the share of time inside the lock.
- Numbers differ from run to run and with machine load. The shape does not: one lock gives about the throughput of one thread; without it, throughput follows the CPUs.
How you meet this in production
- The first symptom is throughput that stops growing while latency rises, with CPU that is either high or oddly low. Adding instances helps and adding threads or CPUs to one instance does not.
- Take two or three thread dumps a few seconds apart with
jcmd <pid> Thread.print. Many threads in BLOCKED (on object monitor) that wait for the same address in every dump point at one lock; the thread that has itlockedshows what is done while holding it. - For a measured answer, record with JFR (
jcmd <pid> JFR.start settings=profile duration=60s filename=/tmp/rec.jfr) and readjfr view contention-by-site. The default settings record monitor waits above 20 ms, the profile settings above 10 ms. - The Investigate case CPU pinned, traffic flat walks through the same kind of evidence.
References
References: Java SE 21 API: ConcurrentHashMap (frequency map with LongAdder, computeIfAbsent) ↗ · Java SE 21 API: LongAdder ↗ · Java SE 21 API: Thread.State (BLOCKED, RUNNABLE) ↗ · JDK 21 tool reference: jcmd (Thread.print, JFR.start, JFR.dump) ↗ · JDK 21 tool reference: jfr (print, view, summary) ↗ · Report a problem