Skip to content
Intermediate20 minSimulation 6 · Java 21

Lock contention: eight threads, the throughput of one

Eight threads share one synchronized map on two CPUs. Throughput stays at the level of one thread and seven threads sit BLOCKED. Find it in a thread dump and in JFR, then remove the shared lock.

The situation

A Spring Boot 3 service counts page views per route in a bean that every request passes through. The pod has 2 CPUs and the load test uses 8 worker threads. Throughput stops at about 563,000 updates per second. Raising the worker count from 8 to 16 changes nothing, and the CPU graph shows about 90% busy, so the team asks for a bigger pod. A thread dump taken during the test shows 7 of the 8 workers in state BLOCKED, all waiting for the same java.util.HashMap.

Read first: Deadlocks, livelocks, starvation and lock contention, Concurrent collections and ConcurrentHashMap, JFR, GC logs and async-profiler

Break it in the simulation

One lock, many threads

Simulation · not a real JVM

Start with 8 threads on 2 CPUs and one global lock. Read the verdict, then move Threads and watch the chart stay flat. Switch Lock to striped to see the CPUs become the limit.

Guided run · 0 of 6 stages done

    1. Set Threads to 1. Leave CPUs at 2, Lock on global and Share of time inside the lock at 70%.

    Look for

    • Throughput is about 566k/s: one thread uses one CPU.
    • Threads at a typical instant shows one box, RUN · lock. Nobody is BLOCKED.
    • CPU utilisation is 50%: the second CPU has nothing to do.
    1. Move Threads to 8, the lab setting.
    2. Then try 16 and 32.

    Look for

    • Throughput stays at about 563k/s. It equals Lock limit, which is half of CPU limit (1,132k/s).
    • The chart Throughput against thread count is flat from 2 threads on.
    • The verdict names the lock as the bottleneck and says more threads will not help.
    1. Set Threads back to 8.
    2. Read the thread boxes, then the simulated Thread dump and the JFR: contention by site line.

    Look for

    • 7 BLOCKED, 1 RUNNABLE: the same count as the real dump from the lab.
    • In the dump, seven threads say waiting to lock <0x00000000f3500540> (a java.util.HashMap) and one says locked with the same address. The address tells you it is one lock.
    • CPU utilisation is about 92% but only half of it is the work of the operations. A busy CPU graph does not prove the CPUs are the limit.
    1. Raise CPUs from 2 to 8.
    2. Put CPUs back to 2 and lower Share of time inside the lock from 70% to 30%.

    Look for

    • More CPUs: Throughput does not move. The lock limit does not depend on the CPU count.
    • A shorter critical section: Lock limit rises above CPU limit and the verdict changes to the CPUs. Doing less inside the lock is a real fix.
    1. Set Share of time inside the lock back to 70% and Threads to 8.
    2. Switch Lock to striped.

    Look for

    • Throughput doubles to about 1,132k/s, the CPU limit.
    • 0 BLOCKED. 2 threads are on a CPU and 6 are RUNNABLE but waiting for a CPU.
    • The dump has no waiting to lock line and the JFR line reports no contention site.
    1. With Lock on striped, raise CPUs step by step to 8.
    2. Then lower Threads to 4.

    Look for

    • Throughput now follows the CPUs: 8 CPUs give about 4,529k/s.
    • With 4 threads on 8 CPUs the verdict names the thread count: now more threads do help.

Wrong, then right

Incorrect implementation

PageViewStats.javaJava
import java.util.HashMap;import java.util.Locale;import java.util.Map;import org.springframework.stereotype.Component;/** Counts page views per route. Called from a HandlerInterceptor on every request. */@Componentpublic class PageViewStats {    private final Map<String, Long> views = new HashMap<>();    public synchronized void record(String path) {        // Everything below runs while holding the one lock of this bean.        String route = path.toLowerCase(Locale.ROOT).replaceAll("/\\d+", "/{id}");        views.merge(route, 1L, Long::sum);    }    public synchronized Map<String, Long> snapshot() {        return new HashMap<>(views);    }}

Why it is written this way

  • synchronized on an instance method locks the bean itself. The bean is a singleton, so every request thread in the service queues for one monitor.
  • The route is normalised with a regular expression while the lock is held. That work touches no shared state, yet it makes the critical section long.
  • It is correct: no update is lost. That is why it passes every test and only shows up under load.
  • Compiled with javac 21 against a stub of @Component; not run inside Spring.

Why it fails

  • A monitor admits one thread at a time. If an update keeps the lock busy for 1.77 µs, the bean can do at most 1 ÷ 1.77 µs = about 563,000 updates per second, on 2 CPUs or on 64.
  • Threads that arrive while the lock is taken go to state BLOCKED. With 8 threads, 7 are waiting at any instant, which is what the dump shows. A ninth thread only makes the queue longer.
  • The CPUs still look busy. In the lab the 8 workers used about 1.8 of 2 CPUs between them (the cpu= and elapsed= fields of the dump), while the completed work was what one CPU can do. The rest goes into trying for the lock and switching threads.
  • Because CPU is high and throughput is flat, the usual reaction is to add CPUs or threads. Neither moves a limit set by one lock.

Corrected implementation

PageViewStats.javaJava
import java.util.Locale;import java.util.Map;import java.util.TreeMap;import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.atomic.LongAdder;import org.springframework.stereotype.Component;/** Counts page views per route. Called from a HandlerInterceptor on every request. */@Componentpublic class PageViewStats {    private final ConcurrentHashMap<String, LongAdder> views = new ConcurrentHashMap<>();    public void record(String path) {        // The expensive part needs no shared state, so it runs with no lock held.        String route = path.toLowerCase(Locale.ROOT).replaceAll("/\\d+", "/{id}");        views.computeIfAbsent(route, k -> new LongAdder()).increment();    }    /** Not one instant in time: counters keep moving while this runs. */    public Map<String, Long> snapshot() {        Map<String, Long> copy = new TreeMap<>();        views.forEach((route, count) -> copy.put(route, count.sum()));        return copy;    }}

Why it is written this way

  • The normalisation runs before any shared structure is touched, so nothing is locked while it runs.
  • computeIfAbsent(...).increment() is the frequency-map idiom from the ConcurrentHashMap documentation. The mapping function only creates a LongAdder, so it is short.
  • LongAdder is built for many threads updating one sum. Its documentation says throughput under high contention is significantly higher than AtomicLong, at the cost of more memory.
  • Compiled with javac 21 against a stub of @Component; not run inside Spring.

Trade-offs

  • snapshot() is no longer one instant in time. LongAdder.sum() is documented as not being an atomic snapshot, and the map can change while it is copied. For statistics that is fine; for money it is not.
  • A LongAdder uses more memory than a Long, and the map still grows with the number of distinct routes. Normalise the keys or cap the map.
  • You cannot update two counters as one atomic step any more. If an invariant spans several keys you need a lock again, held as briefly as possible.
  • The fix moves the limit to the CPUs. At that point throughput is 1,132k/s on 2 CPUs and more threads still do not help; more CPUs or cheaper operations do.

Run it on a real JVM

LockContention.java starts 8 threads that count page views for 5 seconds. With global every update goes through one synchronized HashMap; with striped it uses a ConcurrentHashMap of LongAdder. The same work is done in both, so the difference in updates per second is the cost of the shared lock.

With Docker (CPU and memory limits)

labs/jvm/lock-contentionShell
cd labs/jvm/lock-contentiondocker build -t jvm-lab-lock-contention .# broken: one synchronized HashMapdocker run --rm --cpus=2 --memory=512m jvm-lab-lock-contention LockContention global# fixed: ConcurrentHashMap + LongAdderdocker run --rm --cpus=2 --memory=512m jvm-lab-lock-contention LockContention striped# a 60 second run with a JFR recording, to inspect from a second terminaldocker run --rm --name lock-lab --cpus=2 --memory=512m jvm-lab-lock-contention -XX:StartFlightRecording=filename=/tmp/contention.jfr,settings=profile LockContention global 8 60# second terminal: thread dump, then a copy of the recording and its contention viewdocker exec lock-lab jcmd 1 Thread.printdocker exec lock-lab jcmd 1 JFR.dump filename=/tmp/dump.jfrdocker exec lock-lab jfr view contention-by-site /tmp/dump.jfrdocker exec lock-lab jfr print --events jdk.JavaMonitorEnter /tmp/dump.jfr

Without Docker, on a local JDK 21

labs/jvm/lock-contentionShell
java -XX:ActiveProcessorCount=2 LockContention.java globaljava -XX:ActiveProcessorCount=2 LockContention.java stripedjava -XX:ActiveProcessorCount=2 -XX:StartFlightRecording=filename=/tmp/contention.jfr,settings=profile LockContention.java global 8 60# second terminal; the program prints its pid on the first linejcmd <pid> Thread.printjcmd <pid> JFR.dump filename=/tmp/dump.jfrjfr view contention-by-site /tmp/dump.jfrjfr print --events jdk.JavaMonitorEnter /tmp/dump.jfr
The program: LockContention.java
labs/jvm/lock-contention/LockContention.javaJava
import java.util.HashMap;import java.util.Map;import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.CountDownLatch;import java.util.concurrent.atomic.LongAdder;/** * Lab 6: CPU saturation and lock contention. * * Worker threads count page views per URL for a fixed time. * *   java LockContention.java global   -> one synchronized HashMap: every thread queues for the same lock *   java LockContention.java striped  -> ConcurrentHashMap + LongAdder: threads rarely meet * * Record it:  java -XX:StartFlightRecording=filename=/tmp/contention.jfr,settings=profile LockContention.java global * Read it:    jfr print --events jdk.JavaMonitorEnter /tmp/contention.jfr | head -40 */public class LockContention {    static final Map<String, Long> globalMap = new HashMap<>();    static final Map<String, LongAdder> striped = new ConcurrentHashMap<>();    static void recordGlobal(String url) {        synchronized (globalMap) {            globalMap.merge(url, 1L, Long::sum);            spin(2_000); // a little work done while holding the lock, e.g. formatting a log line        }    }    static void recordStriped(String url) {        striped.computeIfAbsent(url, k -> new LongAdder()).increment();        spin(2_000); // the same work, outside any shared lock    }    static long sink;    static void spin(int n) {        long x = 0;        for (int i = 0; i < n; i++) x += i ^ (x >>> 3);        sink = x;    }    public static void main(String[] args) throws Exception {        String mode = args.length > 0 ? args[0] : "global";        int threads = args.length > 1 ? Integer.parseInt(args[1]) : 8;        int seconds = args.length > 2 ? Integer.parseInt(args[2]) : 5;        System.out.println("pid " + ProcessHandle.current().pid() + ", mode " + mode + ", " + threads + " threads, " + Runtime.getRuntime().availableProcessors() + " CPUs");        LongAdder done = new LongAdder();        CountDownLatch start = new CountDownLatch(1);        long end = System.nanoTime() + seconds * 1_000_000_000L;        Thread[] workers = new Thread[threads];        for (int t = 0; t < threads; t++) {            int id = t;            workers[t] = new Thread(() -> {                try { start.await(); } catch (InterruptedException e) { return; }                int i = 0;                while (System.nanoTime() < end) {                    String url = "/product/" + ((id * 31 + i++) % 200);                    if (mode.equals("global")) recordGlobal(url); else recordStriped(url);                    done.increment();                }            }, "view-counter-" + t);            workers[t].start();        }        start.countDown();        for (Thread w : workers) w.join();        System.out.printf("%s: %,d updates in %d s = %,d updates/s%n", mode, done.sum(), seconds, done.sum() / seconds);    }}
The Dockerfile
labs/jvm/lock-contention/DockerfileDockerfile
# Java 21 lab image. Build:  docker build -t jvm-lab-lock-contention .FROM eclipse-temurin:21-jdkWORKDIR /labCOPY LockContention.java .RUN javac -d out LockContention.java# The JVM reads the container's CPU and memory limits (container support is on by default),# so --cpus and --memory on "docker run" change what Runtime.availableProcessors() and the default heap size report.ENTRYPOINT ["java", "-cp", "out"]CMD ["LockContention"]

$ java -XX:ActiveProcessorCount=2 LockContention.java global

Program outputOne synchronized HashMap

Real JVM output
pid 4547, mode global, 8 threads, 2 CPUs
global: 2,817,166 updates in 5 s = 563,433 updates/s
8 threads on 2 CPUs finish 563,433 updates per second. Keep this number and compare it with the next run.

$ java -XX:ActiveProcessorCount=2 LockContention.java striped

Program outputConcurrentHashMap + LongAdder

Real JVM output
pid 4575, mode striped, 8 threads, 2 CPUs
striped: 5,660,929 updates in 5 s = 1,132,185 updates/s
The same threads, CPUs and work: 1,132,185 updates per second, about twice as many. Two CPUs are now both doing updates.

$ jcmd <pid> Thread.print

Thread dumpThread dump during a global run

Real JVM output
"view-counter-0" #23 [4626] prio=5 os_prio=0 cpu=830.58ms elapsed=3.65s tid=0x00007e8754631690 nid=4626 waiting for monitor entry  [0x00007e872dbfc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at LockContention.recordGlobal(LockContention.java:23)
	- waiting to lock <0x00000000f3500540> (a java.util.HashMap)
	at LockContention.lambda$main$1(LockContention.java:58)
	at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)

"view-counter-1" #24 [4627] prio=5 os_prio=0 cpu=798.40ms elapsed=3.65s tid=0x00007e8754632810 nid=4627 waiting for monitor entry  [0x00007e872dafc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at LockContention.recordGlobal(LockContention.java:23)
	- waiting to lock <0x00000000f3500540> (a java.util.HashMap)
	at LockContention.lambda$main$1(LockContention.java:58)
	at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)

"view-counter-2" #25 [4628] prio=5 os_prio=0 cpu=824.97ms elapsed=3.65s tid=0x00007e87546339d0 nid=4628 waiting for monitor entry  [0x00007e872d9fc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at LockContention.recordGlobal(LockContention.java:23)
	- waiting to lock <0x00000000f3500540> (a java.util.HashMap)
	at LockContention.lambda$main$1(LockContention.java:58)
	at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)

"view-counter-3" #26 [4629] prio=5 os_prio=0 cpu=762.81ms elapsed=3.65s tid=0x00007e8754634b50 nid=4629 waiting for monitor entry  [0x00007e872d8fc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at LockContention.recordGlobal(LockContention.java:23)
	- waiting to lock <0x00000000f3500540> (a java.util.HashMap)
	at LockContention.lambda$main$1(LockContention.java:58)
	at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)

"view-counter-4" #27 [4630] prio=5 os_prio=0 cpu=834.12ms elapsed=3.65s tid=0x00007e8754635d60 nid=4630 waiting for monitor entry  [0x00007e872d7fc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at LockContention.recordGlobal(LockContention.java:23)
	- waiting to lock <0x00000000f3500540> (a java.util.HashMap)
	at LockContention.lambda$main$1(LockContention.java:58)
	at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)

"view-counter-5" #28 [4631] prio=5 os_prio=0 cpu=890.45ms elapsed=3.65s tid=0x00007e87546375f0 nid=4631 runnable  [0x00007e872d6fb000]
   java.lang.Thread.State: RUNNABLE
	at LockContention.spin(LockContention.java:37)
	at LockContention.recordGlobal(LockContention.java:25)
	- locked <0x00000000f3500540> (a java.util.HashMap)
	at LockContention.lambda$main$1(LockContention.java:58)
	at LockContention$Lambda/0x00007e86e01c4208.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)
The excerpt shows view-counter-0 to view-counter-5. Five are BLOCKED (on object monitor), each "waiting to lock <0x00000000f3500540> (a java.util.HashMap)". view-counter-5 is RUNNABLE and has "locked" the same address: it is the owner. In the full file view-counter-6 and 7 are BLOCKED too, so 7 of 8. Also read cpu= against elapsed=: each thread used about 0.8 s of CPU in 3.65 s.

$ jfr view contention-by-site /tmp/contention.jfr

JFRJFR: contention by site

Real JVM output

                               Contention by Site

StackTrace                                                Count    Avg.    Max.
--------------------------------------------------------- ----- ------- -------
LockContention.recordGlobal(String)                         894 22.7 ms 58.7 ms
One line of code accounts for all recorded waits: 894 events from recordGlobal, 22.7 ms on average, 58.7 ms at worst. With settings=profile JFR records a monitor wait only when it lasts longer than 10 ms, so these are the long waits, not all of them.

$ jfr print --events jdk.JavaMonitorEnter /tmp/contention.jfr

JFRJFR: one jdk.JavaMonitorEnter event

Real JVM output
jdk.JavaMonitorEnter {
  startTime = 21:26:35.120 (2026-11-15)
  duration = 12.9 ms
  monitorClass = java.util.HashMap (classLoader = bootstrap)
  previousOwner = "view-counter-1" (javaThreadId = 24)
  address = 0x7E86CC004D10
  eventThread = "view-counter-7" (javaThreadId = 30)
  stackTrace = [
    LockContention.recordGlobal(String) line: 24
    LockContention.lambda$main$1(CountDownLatch, long, int, String, LongAdder) line: 58
    java.lang.Thread.run() line: 1583
    ...
  ]
}
One wait in full: view-counter-7 waited 12.9 ms for a java.util.HashMap monitor that view-counter-1 held before it, and the stack trace gives the line. monitorClass and the top frame are what you search the code for.

$ jfr view hot-methods /tmp/contention.jfr

ProfileJFR: hot methods

Real JVM output

                               Java Methods that Executes the Most

Method                                                                            Samples Percent
--------------------------------------------------------------------------------- ------- -------
LockContention.spin(int)                                                              405  89.01%
LockContention.recordGlobal(String)                                                    12   2.64%
java.lang.Integer.getChars(int, int, byte[])                                           11   2.42%
java.util.HashMap.merge(Object, Object, BiFunction)                                     7   1.54%
LockContention.lambda$main$1(CountDownLatch, long, int, String, LongAdder)              6   1.32%
89% of the execution samples are in spin(), the work done while holding the lock. A CPU profile shows where running threads spend time; it does not show the seven threads that are waiting. That is why you need the dump or the contention view as well.

Real run and simulation: what each one shows

  • The captured files come from OpenJDK 21.0.13 run directly on Linux x86-64 with -XX:ActiveProcessorCount=2, not in Docker. The Docker commands above were not executed when this was written.
  • The real run shows what the model cannot: waits of very different lengths (an average of 22.7 ms and a maximum of 58.7 ms in JFR), because a monitor is not fair and a thread can be passed over many times. The widget only knows averages.
  • In the lab program almost the whole update happens inside the lock. The widget starts at a 70% share and adds a queueing cost to reach the same 563k/s, so its split between the two is an assumption, not a measurement.
  • The simulation shows what one run cannot: the whole curve from 1 to 32 threads, and what happens when you change the CPU count or the share of time inside the lock.
  • Numbers differ from run to run and with machine load. The shape does not: one lock gives about the throughput of one thread; without it, throughput follows the CPUs.

How you meet this in production

  • The first symptom is throughput that stops growing while latency rises, with CPU that is either high or oddly low. Adding instances helps and adding threads or CPUs to one instance does not.
  • Take two or three thread dumps a few seconds apart with jcmd <pid> Thread.print. Many threads in BLOCKED (on object monitor) that wait for the same address in every dump point at one lock; the thread that has it locked shows what is done while holding it.
  • For a measured answer, record with JFR (jcmd <pid> JFR.start settings=profile duration=60s filename=/tmp/rec.jfr) and read jfr view contention-by-site. The default settings record monitor waits above 20 ms, the profile settings above 10 ms.
  • The Investigate case CPU pinned, traffic flat walks through the same kind of evidence.
Investigate · PracticeThe counter is falling behind, and adding threads did not help

References

References: Java SE 21 API: ConcurrentHashMap (frequency map with LongAdder, computeIfAbsent) ↗ · Java SE 21 API: LongAdder ↗ · Java SE 21 API: Thread.State (BLOCKED, RUNNABLE) ↗ · JDK 21 tool reference: jcmd (Thread.print, JFR.start, JFR.dump) ↗ · JDK 21 tool reference: jfr (print, view, summary) ↗ · Report a problem