Skip to content

Path 3 · lesson 6 of 6

Advanced30 minJava 21 (OpenJDK 21.0.13) · Spring Boot 3.5

Deadlocks, livelocks, starvation and lock contention

Four ways threads stop each other making progress, how each one looks in a thread dump and in JFR, and the fix that belongs to each: lock order, timeouts with back-off, shorter critical sections.

What you will learn

  • State the four conditions a deadlock needs and say which one each fix removes
  • Read the Found one Java-level deadlock section of a thread dump and name the two locks and the two code lines involved
  • Tell a deadlock from heavy lock contention by counting BLOCKED threads and finding the owner of the lock they wait for
  • Recognise livelock and retry storms: threads that are RUNNABLE, CPU that is busy, and work that does not finish
  • Measure lock contention with JFR jdk.JavaMonitorEnter events and jfr view contention-by-site
  • Choose between lock ordering, timed tryLock with random back-off, and shrinking or splitting the lock

Before you start

The concept

Locks make threads wait for each other. That is their purpose, and it is also how they fail. There are four distinct failures. They have different symptoms and different fixes, and during an incident the first job is to tell which one you are looking at.

FailureWhat happensThreads look likeCPU
DeadlockTwo or more threads each hold a lock the other needs. None can ever continue.BLOCKED (monitors) or WAITING (parking) (ReentrantLock), for ever. The dump says Found one Java-level deadlock.Idle
LivelockThreads keep reacting to each other, for example taking a lock, failing to get a second, releasing and retrying, and none completes.RUNNABLE or briefly sleeping, in the same retry loop in every dump.Busy
StarvationSome thread could run but never gets the lock or the worker thread it needs, because others always get there first.The victim is BLOCKED, parked or queued for a long time while other threads make progress.Normal
Lock contentionEverything works, slowly. Many threads queue for one lock, so the work runs one thread at a time.Many threads BLOCKED on one address, and one owner that changes from dump to dump.Often busy, sometimes idle

A deadlock needs four conditions at the same time. Remove any one and it cannot happen:

  1. Mutual exclusion. The resource can be held by one thread at a time. A lock is exactly that.
  2. Hold and wait. A thread keeps the locks it has while it waits for another.
  3. No preemption. A lock cannot be taken away from its holder. Only the holder releases it.
  4. Circular wait. There is a cycle: A waits for a lock held by B, and B waits for a lock held by A (or a longer chain that closes).
FixCondition removedCost
One global lock orderCircular waitNothing at run time. A rule that every piece of code taking two of these locks must follow.
tryLock with a timeout, release everything, retryHold and waitCallers must handle failure. Retries need a limit and a random pause, or you trade deadlock for livelock.
One lock instead of twoHold and waitLess concurrency: unrelated work now queues on the same lock.
No shared mutable state: immutable snapshots, atomics, message passingMutual exclusionA different design. Not always possible for multi-step updates.

Why it exists

A deadlock is rare, permanent and quiet. It needs an unlucky interleaving, so it survives testing and appears under production load. When it happens nothing fails: no exception, no log line, no CPU. The threads involved never return, their requests time out at the caller, and every later request that needs either lock joins them. A service with a pool of 200 request threads can lose them one by one over an hour and then stop answering completely. Only a restart ends it.

Lock contention is the opposite: common, gradual and easy to mistake for something else. Latency rises with load, throughput stops growing when threads or CPUs are added, and the usual reaction, a bigger thread pool, makes the queue for the lock longer. In the lab for this lesson eight threads behind one lock did 563,433 updates per second on 2 CPUs; the same work without the shared lock did 1,132,185.

How it works inside

How a deadlock forms

Two transfers run at the same time: one moves money from account A to account B, the other from B to A. Each locks its source account first, then its target. If the first thread locks A and the second locks B before either asks for its second lock, the cycle is closed. Each now waits for a monitor the other holds, and neither will release what it has until it gets what it wants.

Program outputjava Deadlock.java broken (lab deadlock)

Real JVM output
pid 3961, mode broken
ThreadMXBean.findDeadlockedThreads() found 2 threads:
  "transfer-A-to-B" BLOCKED, waiting for Deadlock$Account@71d44a3 held by "transfer-B-to-A"
  "transfer-B-to-A" BLOCKED, waiting for Deadlock$Account@14cd1699 held by "transfer-A-to-B"
The process will now hang. Take a thread dump with: jcmd 3961 Thread.print
The program detects its own deadlock with ThreadMXBean.findDeadlockedThreads(). Each line names a thread, what it waits for and which thread holds that. The two lines point at each other: that is the cycle. The lab forces the unlucky timing with a latch so that it happens on every run. In a service it would need two opposite transfers on the same pair of accounts within the same few microseconds.

Thread dumpjcmd 3961 Thread.print (the two threads, then the end of the dump)

Real JVM output
"transfer-A-to-B" #19 [3981] prio=5 os_prio=0 cpu=0.26ms elapsed=4.84s tid=0x00007cff8863bb70 nid=3981 waiting for monitor entry  [0x00007cff8c732000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at Deadlock.transferBroken(Deadlock.java:27)
	- waiting to lock <0x00000000f7654bc0> (a Deadlock$Account)
	- locked <0x00000000f7654ba0> (a Deadlock$Account)
...
"transfer-B-to-A" #20 [3982] prio=5 os_prio=0 cpu=0.30ms elapsed=4.84s tid=0x00007cff8863cbf0 nid=3982 waiting for monitor entry  [0x00007cff8c632000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at Deadlock.transferBroken(Deadlock.java:27)
	- waiting to lock <0x00000000f7654ba0> (a Deadlock$Account)
	- locked <0x00000000f7654bc0> (a Deadlock$Account)
...
Found one Java-level deadlock:
=============================
"transfer-A-to-B":
  waiting to lock monitor 0x00007cfef0001760 (object 0x00000000f7654bc0, a Deadlock$Account),
  which is held by "transfer-B-to-A"

"transfer-B-to-A":
  waiting to lock monitor 0x00007cfefc0015d0 (object 0x00000000f7654ba0, a Deadlock$Account),
  which is held by "transfer-A-to-B"
...
Java stack information for the threads listed above:
===================================================
"transfer-A-to-B":
	at Deadlock.transferBroken(Deadlock.java:27)
	- waiting to lock <0x00000000f7654bc0> (a Deadlock$Account)
	- locked <0x00000000f7654ba0> (a Deadlock$Account)
...
"transfer-B-to-A":
	at Deadlock.transferBroken(Deadlock.java:27)
	- waiting to lock <0x00000000f7654ba0> (a Deadlock$Account)
	- locked <0x00000000f7654bc0> (a Deadlock$Account)
...
Found 1 deadlock.
Read it from the bottom. Found one Java-level deadlock is printed by the JVM after it has searched for cycles; you do not have to find it yourself. For each thread it gives the lock it waits for and the thread that holds it. Then, under Java stack information, the pair of lines to act on: transfer-A-to-B is waiting to lock <…bc0> and has locked <…ba0>; transfer-B-to-A has the same two addresses the other way round. Both are at Deadlock.java:27, the inner synchronized. The fix is in how that line's two locks are ordered. In the thread list above, both threads are BLOCKED (on object monitor) with cpu=0.26ms and cpu=0.30ms after 4.84 s: they are not doing anything and never will.

The detection covers java.util.concurrent locks as well. A deadlock between two ReentrantLock.lock() calls is reported in the same section, worded differently, and the threads are parked instead of blocked:

Thread dumpjcmd <pid> Thread.print -l (DeadlockTimeouts.java deadlock: the same transfers with ReentrantLock)

Real JVM output
"transfer-A-to-B" #19 [655097] prio=5 os_prio=0 cpu=0.19ms elapsed=2.62s tid=0x00007d17c04b1ef0 nid=655097 waiting on condition  [0x00007d17c4183000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21.0.13/Native Method)
	- parking to wait for  <0x00000000ff4f1b50> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
...
	at java.util.concurrent.locks.ReentrantLock.lock(java.base@21.0.13/ReentrantLock.java:322)
	at DeadlockTimeouts.transferBlocking(DeadlockTimeouts.java:59)
...
   Locked ownable synchronizers:
	- <0x00000000ff4f1b00> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
...
Found one Java-level deadlock:
=============================
"transfer-A-to-B":
  waiting for ownable synchronizer 0x00000000ff4f1b50, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "transfer-B-to-A"

"transfer-B-to-A":
  waiting for ownable synchronizer 0x00000000ff4f1b00, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "transfer-A-to-B"
...
Found 1 deadlock.
The thread is WAITING (parking) and parking to wait for <…b50>, a ReentrantLock$NonfairSync. With -l its Locked ownable synchronizers lists <…b00>, the lock it holds. The deadlock section says waiting for ownable synchronizer … which is held by. A search of your logs or alerts for the word BLOCKED would miss this deadlock. Search for Found one Java-level deadlock instead.

Lock ordering

If every thread takes the locks in the same order, a cycle is impossible: the thread that holds the lower lock is the only one that can be waiting for the higher one. For accounts, order by account id. The transfer from B to A then locks A first, exactly as the transfer from A to B does, and one of them simply waits for the other to finish.

Program outputjava Deadlock.java ordered

Real JVM output
pid 4028, mode ordered
no deadlock: A=1150 B=850 (total 2000)
With the smaller id locked first, both transfers complete: 100 moved from A to B and 250 from B to A, and the total is still 2,000.

tryLock, livelock and retry storms

When no global order exists, for example because the second lock is only known after the first is taken, the alternative is to stop waiting: take the first lock, try the second with tryLock, and if that fails release the first and start again. This removes hold-and-wait. It also creates a new risk. If two threads do this in step, each takes its first lock, fails on the second, releases and retries, for ever. Both are active and neither finishes. That is livelock.

Program outputjava DeadlockTimeouts.java livelock (two opposite transfers, tryLock and retry)

Real JVM output
Each transfer: lock own account, work 1 ms, tryLock(other) without waiting, on failure unlock, back off, retry.
  fixed 1 ms back-off  run 1: A-to-B needed    1 attempts, B-to-A needed    2 attempts, both done after     8 ms (A=1150 B=850)
  fixed 1 ms back-off  run 2: A-to-B needed    2 attempts, B-to-A needed    1 attempts, both done after     8 ms (A=1150 B=850)
  fixed 1 ms back-off  run 3: A-to-B needed    1 attempts, B-to-A needed    2 attempts, both done after     5 ms (A=1150 B=850)
  random 0-4 ms back-off run 1: A-to-B needed    2 attempts, B-to-A needed    1 attempts, both done after     6 ms (A=1150 B=850)
  random 0-4 ms back-off run 2: A-to-B needed    2 attempts, B-to-A needed    1 attempts, both done after     6 ms (A=1150 B=850)
  random 0-4 ms back-off run 3: A-to-B needed    2 attempts, B-to-A needed    1 attempts, both done after    10 ms (A=1150 B=850)
An honest result: on a real JVM two threads did not stay in step. Even with an identical fixed pause, one transfer succeeded on its first or second attempt in every run, because scheduling jitter breaks the symmetry. A lasting two-thread livelock is hard to produce. What does happen in production is the many-thread version below.

Program outputjava -XX:ActiveProcessorCount=2 -Xmx64m DeadlockTimeouts.java retry

Real JVM output
8 threads x 50,000 transfers between 4 accounts, 2 CPUs. Both locks are held for a few microseconds.
  tryLock, retry at once        9,976 ms   failed attempts 6,549,736,066   process CPU 15,300 ms      40,096 transfers/s   total balance 4,000,000
  tryLock, random back-off      1,166 ms   failed attempts     32,540   process CPU    840 ms     343,053 transfers/s   total balance 4,000,000
  lock() in account-id order      711 ms   failed attempts          0   process CPU    680 ms     562,587 transfers/s   total balance 4,000,000
Eight threads, four accounts, 400,000 transfers in all. Retry at once: the transfers took 9,976 ms, with 6.5 billion failed attempts and 15,300 ms of process CPU, both CPUs saturated for the whole run. Threads that could not get a lock spun on tryLock and took CPU time away from the thread that held it. Random back-off: 1,166 ms, 32,540 failed attempts, 840 ms of CPU. Blocking locks in id order: 711 ms, no failed attempts, 680 ms of CPU. All three end with the correct total. No deadlock, and no thread is ever BLOCKED in the first variant: a dump shows only RUNNABLE threads in the retry loop.

Starvation

A starved thread is able to run and never gets its turn. Java's monitors and the default ReentrantLock are not fair: when a lock is released, a thread that is already running can take it ahead of threads that have waited longer. Usually this evens out. It does not when some threads hold the lock for long periods and re-acquire it at once, or when one kind of work always arrives before another.

Program outputjava -XX:ActiveProcessorCount=2 -Xmx128m LocksBench.java (last part)

Real JVM output
Fairness: 8 threads take one ReentrantLock in a loop for 1 s
  new ReentrantLock()            26,679,730 acquisitions/s   fewest per thread  3,161,952   most per thread  3,791,635
  new ReentrantLock(true)         1,470,684 acquisitions/s   fewest per thread    132,452   most per thread    259,779
Eight identical threads take one lock in a loop for a second. With the default lock every thread got between 3.2 and 3.8 million acquisitions: no starvation here, although nothing guarantees it. A fair lock (new ReentrantLock(true)) guarantees the order of arrival and did 18 times fewer acquisitions in total. Fairness is a cure for a measured starvation problem, and an expensive one.
  • Writer starvation on a read-write lock: a steady stream of readers keeps the read lock held, and a writer waits for a gap that never comes. See the locks lesson.
  • Thread-pool starvation: every worker waits for a task that is queued behind them on the same pool. No lock is involved and no deadlock is reported. See CompletableFuture starvation.
  • Priority and greedy threads: one thread calls a long synchronized method in a tight loop, and others that need the same monitor are blocked most of the time.

Lock contention

Contention is what a lock costs when it is busy. If each operation spends a fraction of its time inside one shared lock, that fraction runs one thread at a time whatever the number of threads and CPUs. If 70% of each operation is inside the lock, throughput can never exceed about 1.4 times that of a single thread. Threads beyond that only lengthen the queue. Each handover also costs: a blocked thread has to be woken and scheduled before it can use the lock.

Program outputjava LockContention.java global and java LockContention.java striped (lab lock-contention)

Real JVM output
pid 4547, mode global, 8 threads, 2 CPUs
global: 2,817,166 updates in 5 s = 563,433 updates/s
pid 4575, mode striped, 8 threads, 2 CPUs
striped: 5,660,929 updates in 5 s = 1,132,185 updates/s
The same work on the same 2 CPUs with the same 8 threads. With one synchronized HashMap, 563,433 updates per second. With a ConcurrentHashMap of LongAdders and the extra work moved outside any shared lock, 1,132,185: twice as much.

Thread dumpjcmd 4603 Thread.print (LockContention global: three of the eight workers)

Real JVM output
"view-counter-0" #23 [4626] prio=5 os_prio=0 cpu=830.58ms elapsed=3.65s tid=0x00007e8754631690 nid=4626 waiting for monitor entry  [0x00007e872dbfc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at LockContention.recordGlobal(LockContention.java:23)
	- waiting to lock <0x00000000f3500540> (a java.util.HashMap)
...
"view-counter-1" #24 [4627] prio=5 os_prio=0 cpu=798.40ms elapsed=3.65s tid=0x00007e8754632810 nid=4627 waiting for monitor entry  [0x00007e872dafc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at LockContention.recordGlobal(LockContention.java:23)
	- waiting to lock <0x00000000f3500540> (a java.util.HashMap)
...
"view-counter-5" #28 [4631] prio=5 os_prio=0 cpu=890.45ms elapsed=3.65s tid=0x00007e87546375f0 nid=4631 runnable  [0x00007e872d6fb000]
   java.lang.Thread.State: RUNNABLE
	at LockContention.spin(LockContention.java:37)
	at LockContention.recordGlobal(LockContention.java:25)
	- locked <0x00000000f3500540> (a java.util.HashMap)
In the full dump seven of the eight view-counter threads are BLOCKED (on object monitor), all waiting to lock <0x00000000f3500540>, a java.util.HashMap. The eighth, view-counter-5, is RUNNABLE and its stack has - locked <0x00000000f3500540>: it is the owner, and it is at LockContention.spin, doing work inside the lock. There is no deadlock section. Look also at cpu=830.58ms elapsed=3.65s: this thread was on a CPU for less than a quarter of its life. Take a second dump and the owner will be a different thread: that is contention. In a deadlock the same threads hold the same locks in every dump.

JFRjfr print --events jdk.JavaMonitorEnter and jfr view contention-by-site (same run, settings=profile)

Real JVM output
jdk.JavaMonitorEnter {
  startTime = 21:26:35.120 (2026-11-15)
  duration = 12.9 ms
  monitorClass = java.util.HashMap (classLoader = bootstrap)
  previousOwner = "view-counter-1" (javaThreadId = 24)
  address = 0x7E86CC004D10
  eventThread = "view-counter-7" (javaThreadId = 30)
  stackTrace = [
    LockContention.recordGlobal(String) line: 24
    LockContention.lambda$main$1(CountDownLatch, long, int, String, LongAdder) line: 58
    java.lang.Thread.run() line: 1583
    ...
  ]
}

                               Contention by Site

StackTrace                                                Count    Avg.    Max.
--------------------------------------------------------- ----- ------- -------
LockContention.recordGlobal(String)                         894 22.7 ms 58.7 ms
One jdk.JavaMonitorEnter event is one wait to enter a monitor. duration = 12.9 ms is how long view-counter-7 waited, monitorClass is the class of the locked object, previousOwner is the thread that held it, and the stack gives the line that wanted the lock: LockContention.recordGlobal(String) line: 24. The view below it adds them up: 894 waits at that one site, 22.7 ms on average and 58.7 ms at worst, for an operation that takes microseconds. Only waits of at least 10 ms are recorded with settings=profile (20 ms with the default settings), so the true number of waits is higher.

Architecture

Drawing diagram…

Flowchart. Take a thread dump with lock information. If it ends with a deadlock section, it is a deadlock: restart and fix the lock order. Otherwise, if many threads wait on one lock address, check the owner: an owner that changes between dumps means contention, the same owner in a slow call means a lock held across input or output, and no owner means a leaked or read lock or a non-lock wait. If few threads wait but CPU is high with threads in a retry loop, it is a livelock or retry storm. Otherwise look at pool starvation, garbage collection or a slow dependency.

flowchart TD
  A["Requests hang or are slow"] --> B["Take a thread dump with jcmd Thread.print -l"]
  B --> C{"Does it end with Found one Java-level deadlock"}
  C -->|"Yes"| D["Deadlock. Restart, then fix the lock order at the lines shown"]
  C -->|"No"| E{"Are many threads BLOCKED or parked on one address"}
  E -->|"Yes"| F{"Is there an owner, and does it change between dumps"}
  F -->|"Owner changes"| G["Lock contention. Shorten or split the critical section"]
  F -->|"Same owner, in a slow call"| H["Lock held across I/O. Move the call out of the lock"]
  F -->|"No owner listed"| I["Leaked or read lock, or a non-lock wait. Not detectable as deadlock"]
  E -->|"No"| J{"Is CPU high with threads RUNNABLE in a retry loop"}
  J -->|"Yes"| K["Livelock or retry storm. Add random back-off and a retry limit"]
  J -->|"No"| L["Look elsewhere, such as pool starvation, GC or a slow dependency"]
From a hung or slow service to the kind of lock problem, using two thread dumps a few seconds apart.
  • The deadlock section is the first thing to check because it is unambiguous and printed by the JVM itself. Its absence rules out only cycles of owned locks.
  • For everything else you need two dumps, 5 to 10 seconds apart. One dump cannot tell a thread that is stuck from a thread that happened to be there.
  • Group waiting threads by the address in waiting to lock <…> or parking to wait for <…>. One address with many waiters is one lock. Then find the thread whose stack has - locked <same address>, or which lists it under Locked ownable synchronizers.
  • The livelock branch has no waiting threads at all. The evidence is CPU time and the same RUNNABLE frames in every dump, with counters that show retries rising and completions flat.

Step by step

Drawing diagram…

Sequence diagram. In the first half, the transfer from A to B locks account A while the transfer from B to A locks account B. Each then asks for the other account and blocks while holding its own, which closes a cycle. In the second half both transfers ask for account A first. The second one waits, the first takes B, finishes and unlocks, and the second continues.

sequenceDiagram
  participant T1 as transfer A to B
  participant A as Account A lock
  participant B as Account B lock
  participant T2 as transfer B to A
  T1->>A: lock, acquired
  T2->>B: lock, acquired
  T1->>B: lock, held by the other transfer
  Note over T1: BLOCKED, still holding A
  T2->>A: lock, held by the other transfer
  Note over T2: BLOCKED, still holding B
  Note over T1,T2: Cycle closed. Neither will ever release
  Note over A,B: With one global order both ask for A first
  T1->>A: lock, acquired
  T2->>A: lock, waits its turn
  T1->>B: lock, acquired
  T1->>A: unlock both
  A-->>T2: acquired, continues to B
The same two transfers twice: locks taken in argument order, then in one global order.
  1. The deadlock needs both first locks to be taken before either second lock is requested. That window is a few instructions wide, which is why the bug can sit in production for months.
  2. After the cycle closes, nothing that happens later can open it. Timeouts at the HTTP client free the caller, not these two threads.
  3. With one order, the second transfer waits before it holds anything. A thread that holds nothing cannot be part of a cycle.
  4. The ordering rule is about the locks, not the direction of the business operation. The transfer from B to A still debits B and credits A.

In a real system

A wallet service keeps balances in memory and moves money between accounts. Once every few weeks, a handful of requests never return. The count of busy request threads climbs by two at a time and never comes down, and after some days the instance is restarted for an unrelated deploy and the problem resets. This is the usual shape of a production deadlock: small, permanent, and found late. The Investigate case Requests that never return walks through one.

  • First action. Take a thread dump before restarting: jcmd <pid> Thread.print -l > /tmp/dump-1.txt. A restart destroys the only evidence. Then restart: a deadlock does not heal.
  • Reading it. Go to the end of the file. If Found one Java-level deadlock is there, the two waiting to lock / locked pairs under Java stack information name the lines to fix.
  • Finding it earlier. A scheduled check with ThreadMXBean.findDeadlockedThreads() can log or alert within seconds. The documentation says the method is meant for troubleshooting and might be expensive, so run it every 30 seconds, not on every request.
  • Preventing it. Keep the code that takes two locks in one place and order by a stable key. Do not call code you do not control, such as listeners or callbacks, while holding a lock: it may take locks of its own in an order you cannot see.

Contention is far more common. A product service counts page views in one synchronized map. At normal traffic nobody notices. During a sale, p99 latency rises from 20 ms to 400 ms while CPU is busy and the downstream systems are healthy. Adding pods helps, because each pod has its own lock; doubling the threads per pod does nothing. Two dumps show most request threads BLOCKED on one java.util.HashMap with a different owner each time, and JFR shows hundreds of jdk.JavaMonitorEnter events at one line. The Investigate case CPU pinned with flat traffic covers this pattern.

You seeIt isDo
Found one Java-level deadlock at the end of the dumpDeadlockSave the dump, restart, impose one lock order at the two lines named
Many threads BLOCKED on one address, owner RUNNABLE and different in each dumpContentionShorten the critical section, split the lock, or use a concurrent structure
Many threads waiting on one address, same owner each time, owner in a socket or JDBC callLock held across a slow callMove the call outside the lock; add a timeout to the call
CPU saturated, threads RUNNABLE in tryLock or a CAS loop, throughput near zeroLivelock or retry stormRandom back-off, a retry limit, or blocking locks in a fixed order
One kind of request times out while others are fast; its threads wait on a lock others hold in long loopsStarvationShorten the long holder; as a last resort a fair lock

Java and Spring Boot

Two Spring Boot classes for the wallet service: a transfer that uses timed tryLock with random back-off, for situations where a lock order cannot be imposed, and a watchdog that reports deadlocks. They were compiled with javac on JDK 21 against stand-ins for the Spring and SLF4J types, not run inside a Spring application. The lock-ordering version is in the next section.

src/main/java/com/example/wallet/TimedTransferService.javaJava
package com.example.wallet;import java.util.concurrent.ThreadLocalRandom;import java.util.concurrent.TimeUnit;import org.springframework.stereotype.Service;/** For code that cannot impose one lock order, for example when the second lock is only known after the first is held. */@Servicepublic class TimedTransferService {    private static final int MAX_ATTEMPTS = 5;    private static final long LOCK_WAIT_MILLIS = 50;    private final AccountRepository accounts;    public TimedTransferService(AccountRepository accounts) {        this.accounts = accounts;    }    public void transfer(long fromId, long toId, long cents) throws InterruptedException {        Account from = accounts.get(fromId);        Account to = accounts.get(toId);        for (int attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {            if (from.lock().tryLock(LOCK_WAIT_MILLIS, TimeUnit.MILLISECONDS)) {                try {                    if (to.lock().tryLock(LOCK_WAIT_MILLIS, TimeUnit.MILLISECONDS)) {                        try {                            if (from.balanceCents() < cents) {                                throw new InsufficientFundsException(fromId);                            }                            from.add(-cents);                            to.add(cents);                            return;                        } finally {                            to.lock().unlock();                        }                    }                } finally {                    from.lock().unlock();          // gave up on the second lock: release the first as well                }            }            // Wait a random time before trying again, so two transfers that collided do not collide again in step.            TimeUnit.MILLISECONDS.sleep(ThreadLocalRandom.current().nextLong(5, 25L * attempt));        }        throw new TransferBusyException("could not lock accounts " + fromId + " and " + toId + " after " + MAX_ATTEMPTS + " attempts");    }}

Why it is written this way

  • Both acquisitions have a time limit, so no thread can wait for ever. That removes hold-and-wait: a thread that cannot get the second lock gives the first one back.
  • Each unlock() is in a finally that belongs to the tryLock that succeeded. The outer finally runs whether or not the inner tryLock got the lock.
  • The pause is random and grows with the attempt number. A fixed pause, or none, lets colliding threads collide again; in the measurement above, retrying at once cost 6.5 billion failed attempts.
  • MAX_ATTEMPTS turns an unlucky transfer into an error the caller can see and retry, instead of a thread that spins without limit. A controller would map TransferBusyException to 503.
  • Account holds an id, a balance and a ReentrantLock; AccountRepository returns the one Account object per id. Both are assumed from the rest of the project.

Wrong, then right

The transfer method of the wallet service. The incorrect version locks the two accounts in the order of its arguments, which is the natural way to write it and is correct for any single transfer.

Incorrect implementation

src/main/java/com/example/wallet/TransferService.javaJava
package com.example.wallet;import org.springframework.stereotype.Service;@Servicepublic class TransferService {    private final AccountRepository accounts;   // in-memory ledger: one Account object per account id    public TransferService(AccountRepository accounts) {        this.accounts = accounts;    }    public void transfer(long fromId, long toId, long cents) {        Account from = accounts.get(fromId);        Account to = accounts.get(toId);        synchronized (from) {                    // lock order = argument order            synchronized (to) {                if (from.balanceCents() < cents) {                    throw new InsufficientFundsException(fromId);                }                from.add(-cents);                to.add(cents);            }        }    }}

Why it is written this way

  • The lock order depends on the caller's arguments. transfer(1, 2, …) locks account 1 then 2; transfer(2, 1, …) locks 2 then 1.
  • While waiting for to, the thread keeps from locked. synchronized has no timeout and cannot be interrupted, so the wait has no end.

Why it fails

  • Two requests move money between the same two accounts in opposite directions at the same moment. Each thread enters its outer synchronized, then blocks on the inner one, which the other thread holds. All four deadlock conditions hold: exclusive locks, hold and wait, no preemption, and a cycle of two.
  • Neither thread ever returns. In the real run the dump shows both BLOCKED (on object monitor) at the inner synchronized, with about 0.3 ms of CPU time each after 4.84 s, and the JVM prints Found one Java-level deadlock.
  • The damage spreads. Both accounts stay locked, so every later transfer or balance read that touches either account blocks behind them. Request threads are lost one by one, the callers see timeouts with no server-side error, and CPU stays idle so autoscaling does nothing.
  • It cannot be reproduced on demand. A single-threaded test passes, and a load test hits it only if two opposite transfers on one pair of accounts overlap within microseconds.

Corrected implementation

src/main/java/com/example/wallet/TransferService.javaJava
package com.example.wallet;import org.springframework.stereotype.Service;@Servicepublic class TransferService {    private final AccountRepository accounts;   // in-memory ledger: one Account object per account id    public TransferService(AccountRepository accounts) {        this.accounts = accounts;    }    public void transfer(long fromId, long toId, long cents) {        if (fromId == toId) {            throw new IllegalArgumentException("source and target are the same account");        }        Account from = accounts.get(fromId);        Account to = accounts.get(toId);        // One global order for every thread: the account with the smaller id is always locked first,        // whichever direction the money moves.        Account first = from.id() < to.id() ? from : to;        Account second = first == from ? to : from;        synchronized (first) {            synchronized (second) {                if (from.balanceCents() < cents) {                    throw new InsufficientFundsException(fromId);                }                from.add(-cents);                to.add(cents);            }        }    }}

Why it is written this way

  • first is always the account with the smaller id. Two opposite transfers now both ask for the same account first, so one waits before it holds anything.
  • The business direction is unchanged: money still leaves from and arrives at to. Only the order of locking is fixed.
  • The same-account check is needed for correctness of the rule: with equal ids there is no order, and the transfer would be meaningless anyway.
  • The id must be stable and unique. Ordering by System.identityHashCode is a fallback when there is no key, and needs a tie-breaking lock for the rare equal hash.

Trade-offs

  • The rule is global and nothing enforces it. Every method that locks two accounts must use the same order, and one new method that locks in argument order brings the deadlock back. Keep all two-account locking in one helper.
  • It covers only these locks. If code inside the block takes another lock, calls a listener, or waits for a database row lock, a cycle can still form through that resource, and the JVM will not see the database part of it.
  • A transfer between two busy accounts now waits its turn with no time limit. That is correct, but a slow holder still delays everyone behind it; ordering prevents deadlock, not contention.
  • In-memory locks protect one JVM. With several instances of the service, the same two accounts can be changed on two machines at once, and the ordering has to move to where the data is, for example row locks taken in id order inside one database transaction.

Configuration

Deadlocks have no setting. What can be configured is what the diagnostic tools show. Defaults are those of JDK 21, read from the tool reference and from lib/jfr/default.jfc and profile.jfc in the JDK.

jcmd <pid> Thread.print -lDefault -l is off
What it does
Adds Locked ownable synchronizers to each thread: the ReentrantLocks and write locks it holds. Monitors held are always shown as - locked <…> in the stack.
When to change it
Use -l whenever threads are parking to wait for a lock. The deadlock section is printed with or without it.
jdk.JavaMonitorEnter#threshold (JFR)Default 20 ms in default.jfc, 10 ms in profile.jfc
What it does
Waits to enter a synchronized block shorter than this are not recorded.
When to change it
For a short investigation lower it, for example jcmd <pid> JFR.start settings=profile jdk.JavaMonitorEnter#threshold=1ms duration=60s filename=/tmp/locks.jfr. Many short waits can cost more in total than a few long ones.
jdk.ThreadPark#threshold (JFR)Default 20 ms in default.jfc, 10 ms in profile.jfc
What it does
The matching event for waits on java.util.concurrent locks, which do not produce jdk.JavaMonitorEnter.
When to change it
Include it when the contended lock is a ReentrantLock. jfr view contention-by-site counts monitor waits only.
new ReentrantLock(boolean fair)Default false (non-fair)
What it does
A fair lock is granted in order of arrival, which prevents starvation of waiting threads.
When to change it
Turn it on only for a measured starvation problem. In this lesson's run a fair lock did 18 times fewer acquisitions per second.

Hands-on exercise

Build a deadlock, read it, and remove one of its conditions

Step two opposite transfers into a deadlock, read the wait-for cycle and the simulated dump, then prevent it first with a global lock order and then with a tryLock timeout.

Steps · 0 of 6 done

Two transfers, two locks

Simulation · not a real JVM

A simulation of the lab program. You decide which thread takes the next step, so you can produce the unlucky interleaving on purpose, then change one rule and try again.

Take it further in the simulation: Deadlock: two transfers, two locks, opposite order

What can go wrong

Lock order taken from the arguments
Symptom
Rarely, two requests never return and the threads serving them stay BLOCKED for ever. The dump ends with Found one Java-level deadlock.
Cause
Two code paths, or one path called with swapped arguments, take the same two locks in opposite orders.
Fix
Order the locks by a stable unique key before taking them, in one shared helper. Review every place that holds one of these locks while taking another.
Calling foreign code while holding a lock
Symptom
A deadlock whose second lock is in a library, a listener, a logger appender or a cache loader that your code never mentions.
Cause
A callback invoked inside a synchronized block takes its own lock, while another thread holds that lock and calls back into your class.
Fix
Copy what the callback needs while holding the lock, release the lock, then call out. Keep critical sections free of calls you do not control.
tryLock retry without back-off
Symptom
CPU at 100%, throughput close to zero, no BLOCKED threads, no deadlock section. Dumps show the same threads RUNNABLE in a loop around tryLock.
Cause
Failed attempts retry at once. Spinning threads take CPU time from the thread that holds the lock, so the lock is held longer and more attempts fail.
Fix
Wait a random time between attempts, limit the attempts, and fail visibly after the limit. Where a lock order exists, prefer it: it needs no retries at all.
Reading many BLOCKED threads as a deadlock
Symptom
A team reorders locks or restarts the service after seeing 150 BLOCKED threads. The slowness returns under the next load peak.
Cause
The threads were queueing behind an owner that was doing work. There was no cycle, so there was no deadlock to fix.
Fix
Check for the deadlock section first. Without it, find the owner and what it does while holding the lock, and compare two dumps a few seconds apart.
Adding threads to a contended lock
Symptom
The pool is raised from 50 to 200 threads. Throughput does not improve, latency gets worse, and more threads show as BLOCKED.
Cause
The part of each request inside the lock runs one thread at a time. More threads only add waiters and context switches.
Fix
Reduce the time inside the lock, or the sharing: move work out of the critical section, use ConcurrentHashMap and LongAdder, or split one lock into several.
Restarting before taking a dump
Symptom
The hang comes back every few weeks and nobody knows why. Each time the instance was restarted to restore service.
Cause
The thread dump is the only record of which threads and locks formed the cycle, and a restart erases it.
Fix
Make jcmd <pid> Thread.print -l to a file part of the restart procedure, or run a watchdog that logs findDeadlockedThreads().

How to troubleshoot

Some requests never return and CPU is idle

Check

jcmd <pid> Thread.print -l > /tmp/dump-1.txt   then   grep -n -A12 'Found one Java-level deadlock' /tmp/dump-1.txt
How to read it
If the section exists, it lists each thread, the lock it waits for and the thread holding it. The stacks below it show the two lines with waiting to lock and locked. Those lines take the locks in opposite orders.
Service is slow under load and many threads are BLOCKED

Check

grep -c 'java.lang.Thread.State: BLOCKED' /tmp/dump-1.txt   and   grep 'waiting to lock' /tmp/dump-1.txt | sort | uniq -c | sort -rn | head
How to read it
The first number is how many threads wait for monitors. The second groups them by lock. One address with most of the waiters is the contended lock. In the lab dump it is 7 threads on one java.util.HashMap.
Need the owner of the contended monitor

Check

grep -n -B12 -e '- locked <0x00000000f3500540>' /tmp/dump-1.txt   (use the address from the previous step)
How to read it
The thread whose stack contains - locked <that address> owns it. RUNNABLE in application code means work inside the lock. A socket or database frame means a slow call inside the lock. If the owner is itself BLOCKED on another lock, follow the chain.
Need to know which lock costs the most time

Check

jcmd <pid> JFR.start settings=profile duration=60s filename=/tmp/locks.jfr   then   jfr view contention-by-site /tmp/locks.jfr   and   jfr print --events jdk.JavaMonitorEnter /tmp/locks.jfr
How to read it
The view lists each call site with the number of waits and their average and maximum duration. Count times average is the time lost at that site. In the lab run: 894 waits averaging 22.7 ms at LockContention.recordGlobal.
CPU is saturated, throughput is near zero, nothing is BLOCKED

Check

Take three dumps a second apart with jcmd <pid> Thread.print and compare the top frames of the RUNNABLE application threads; top -H -p <pid> shows which threads use the CPU
How to read it
The same threads in the same retry loop (tryLock, compareAndSet) in every dump, all using CPU, is a livelock or retry storm. Threads that are RUNNABLE in different places each time are doing real work.
Want to detect a deadlock from inside the application

Check

ManagementFactory.getThreadMXBean().findDeadlockedThreads()   (null means none)
How to read it
A non-null array holds the ids of the threads in the cycle. Pass it to getThreadInfo(ids, true, true) for names, the lock each waits for and its owner. Cycles that include virtual threads are not found.

Key takeaways

  • A deadlock needs exclusive locks, hold and wait, no preemption and a cycle. One global lock order removes the cycle and costs nothing at run time.
  • The JVM finds deadlocks for you: the dump ends with Found one Java-level deadlock, and the waiting to lock and locked pairs name the lines. It covers ReentrantLock too, as ownable synchronizer.
  • A deadlock never heals. Save a dump, then restart.
  • Many BLOCKED threads without that section is contention. Find the owner of the address they wait for and look at what it does inside the lock.
  • tryLock with a timeout removes hold-and-wait but needs random back-off and a retry limit. Retrying at once burned 15 s of CPU for work that needed 0.7 s.
  • Contention makes part of every request run one thread at a time. More threads lengthen the queue; shrinking or splitting the lock doubled throughput in the lab.
  • JFR measures it: jdk.JavaMonitorEnter events and jfr view contention-by-site for monitors, jdk.ThreadPark for java.util.concurrent locks.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A pricing service is slow. A dump shows 180 of 200 request threads BLOCKED, all 'waiting to lock <0x00000007a1c0e2f8> (a com.example.PriceCache)'. One thread is RUNNABLE in SocketInputStream.read with '- locked <0x00000007a1c0e2f8>' further down its stack. The dump has no deadlock section. A second dump 10 s later shows the same owner in the same frame.

What is the problem and the fitting fix?

Scenario 2 of 3

An order service hangs partially every few weeks. A dump taken during the hang ends with 'Found one Java-level deadlock': thread http-17 is at OrderService.reserve and waits for an Inventory monitor while it has locked an Order; thread http-44 is at InventoryService.restock and waits for that Order while it has locked the Inventory.

What should the team do?

Scenario 3 of 3

To avoid deadlocks a team replaced nested synchronized blocks with a loop: lock the first account, tryLock() the second, and on failure unlock and retry at once. In a load test with 64 threads, CPU is at 100% on all cores, completed transfers per second fall to a tenth, no thread is BLOCKED, and the dump has no deadlock section.

What explains the result and what should change?

References

References: Java SE 21 API: ThreadMXBean (findDeadlockedThreads, monitors and ownable synchronizers, virtual threads not covered) ↗ · JDK 21 tool reference: jcmd (Thread.print and its -l option, JFR.start) ↗ · JDK 21 tool reference: jfr (print, summary, view) ↗ · Java SE 21 API: Thread.State (BLOCKED, WAITING, TIMED_WAITING) ↗ · Java SE 21 API: ReentrantLock (tryLock with timeout, fairness and its throughput cost) ↗ · The Java Tutorials: Deadlock ↗ · The Java Tutorials: Starvation and Livelock ↗ · Report a problem