Path 3 · lesson 6 of 6
Deadlocks, livelocks, starvation and lock contention
Four ways threads stop each other making progress, how each one looks in a thread dump and in JFR, and the fix that belongs to each: lock order, timeouts with back-off, shorter critical sections.
What you will learn
- State the four conditions a deadlock needs and say which one each fix removes
- Read the
Found one Java-level deadlocksection of a thread dump and name the two locks and the two code lines involved - Tell a deadlock from heavy lock contention by counting
BLOCKEDthreads and finding the owner of the lock they wait for - Recognise livelock and retry storms: threads that are
RUNNABLE, CPU that is busy, and work that does not finish - Measure lock contention with JFR
jdk.JavaMonitorEnterevents andjfr view contention-by-site - Choose between lock ordering, timed
tryLockwith random back-off, and shrinking or splitting the lock
Before you start
The concept
Locks make threads wait for each other. That is their purpose, and it is also how they fail. There are four distinct failures. They have different symptoms and different fixes, and during an incident the first job is to tell which one you are looking at.
| Failure | What happens | Threads look like | CPU |
|---|---|---|---|
| Deadlock | Two or more threads each hold a lock the other needs. None can ever continue. | BLOCKED (monitors) or WAITING (parking) (ReentrantLock), for ever. The dump says Found one Java-level deadlock. | Idle |
| Livelock | Threads keep reacting to each other, for example taking a lock, failing to get a second, releasing and retrying, and none completes. | RUNNABLE or briefly sleeping, in the same retry loop in every dump. | Busy |
| Starvation | Some thread could run but never gets the lock or the worker thread it needs, because others always get there first. | The victim is BLOCKED, parked or queued for a long time while other threads make progress. | Normal |
| Lock contention | Everything works, slowly. Many threads queue for one lock, so the work runs one thread at a time. | Many threads BLOCKED on one address, and one owner that changes from dump to dump. | Often busy, sometimes idle |
A deadlock needs four conditions at the same time. Remove any one and it cannot happen:
- Mutual exclusion. The resource can be held by one thread at a time. A lock is exactly that.
- Hold and wait. A thread keeps the locks it has while it waits for another.
- No preemption. A lock cannot be taken away from its holder. Only the holder releases it.
- Circular wait. There is a cycle: A waits for a lock held by B, and B waits for a lock held by A (or a longer chain that closes).
| Fix | Condition removed | Cost |
|---|---|---|
| One global lock order | Circular wait | Nothing at run time. A rule that every piece of code taking two of these locks must follow. |
tryLock with a timeout, release everything, retry | Hold and wait | Callers must handle failure. Retries need a limit and a random pause, or you trade deadlock for livelock. |
| One lock instead of two | Hold and wait | Less concurrency: unrelated work now queues on the same lock. |
| No shared mutable state: immutable snapshots, atomics, message passing | Mutual exclusion | A different design. Not always possible for multi-step updates. |
Why it exists
A deadlock is rare, permanent and quiet. It needs an unlucky interleaving, so it survives testing and appears under production load. When it happens nothing fails: no exception, no log line, no CPU. The threads involved never return, their requests time out at the caller, and every later request that needs either lock joins them. A service with a pool of 200 request threads can lose them one by one over an hour and then stop answering completely. Only a restart ends it.
Lock contention is the opposite: common, gradual and easy to mistake for something else. Latency rises with load, throughput stops growing when threads or CPUs are added, and the usual reaction, a bigger thread pool, makes the queue for the lock longer. In the lab for this lesson eight threads behind one lock did 563,433 updates per second on 2 CPUs; the same work without the shared lock did 1,132,185.
How it works inside
How a deadlock forms
Two transfers run at the same time: one moves money from account A to account B, the other from B to A. Each locks its source account first, then its target. If the first thread locks A and the second locks B before either asks for its second lock, the cycle is closed. Each now waits for a monitor the other holds, and neither will release what it has until it gets what it wants.
Program outputjava Deadlock.java broken (lab deadlock)
Real JVM outputpid 3961, mode broken ThreadMXBean.findDeadlockedThreads() found 2 threads: "transfer-A-to-B" BLOCKED, waiting for Deadlock$Account@71d44a3 held by "transfer-B-to-A" "transfer-B-to-A" BLOCKED, waiting for Deadlock$Account@14cd1699 held by "transfer-A-to-B" The process will now hang. Take a thread dump with: jcmd 3961 Thread.print
ThreadMXBean.findDeadlockedThreads(). Each line names a thread, what it waits for and which thread holds that. The two lines point at each other: that is the cycle. The lab forces the unlucky timing with a latch so that it happens on every run. In a service it would need two opposite transfers on the same pair of accounts within the same few microseconds.Thread dumpjcmd 3961 Thread.print (the two threads, then the end of the dump)
Real JVM output"transfer-A-to-B" #19 [3981] prio=5 os_prio=0 cpu=0.26ms elapsed=4.84s tid=0x00007cff8863bb70 nid=3981 waiting for monitor entry [0x00007cff8c732000] java.lang.Thread.State: BLOCKED (on object monitor) at Deadlock.transferBroken(Deadlock.java:27) - waiting to lock <0x00000000f7654bc0> (a Deadlock$Account) - locked <0x00000000f7654ba0> (a Deadlock$Account) ... "transfer-B-to-A" #20 [3982] prio=5 os_prio=0 cpu=0.30ms elapsed=4.84s tid=0x00007cff8863cbf0 nid=3982 waiting for monitor entry [0x00007cff8c632000] java.lang.Thread.State: BLOCKED (on object monitor) at Deadlock.transferBroken(Deadlock.java:27) - waiting to lock <0x00000000f7654ba0> (a Deadlock$Account) - locked <0x00000000f7654bc0> (a Deadlock$Account) ... Found one Java-level deadlock: ============================= "transfer-A-to-B": waiting to lock monitor 0x00007cfef0001760 (object 0x00000000f7654bc0, a Deadlock$Account), which is held by "transfer-B-to-A" "transfer-B-to-A": waiting to lock monitor 0x00007cfefc0015d0 (object 0x00000000f7654ba0, a Deadlock$Account), which is held by "transfer-A-to-B" ... Java stack information for the threads listed above: =================================================== "transfer-A-to-B": at Deadlock.transferBroken(Deadlock.java:27) - waiting to lock <0x00000000f7654bc0> (a Deadlock$Account) - locked <0x00000000f7654ba0> (a Deadlock$Account) ... "transfer-B-to-A": at Deadlock.transferBroken(Deadlock.java:27) - waiting to lock <0x00000000f7654ba0> (a Deadlock$Account) - locked <0x00000000f7654bc0> (a Deadlock$Account) ... Found 1 deadlock.
Found one Java-level deadlock is printed by the JVM after it has searched for cycles; you do not have to find it yourself. For each thread it gives the lock it waits for and the thread that holds it. Then, under Java stack information, the pair of lines to act on: transfer-A-to-B is waiting to lock <…bc0> and has locked <…ba0>; transfer-B-to-A has the same two addresses the other way round. Both are at Deadlock.java:27, the inner synchronized. The fix is in how that line's two locks are ordered. In the thread list above, both threads are BLOCKED (on object monitor) with cpu=0.26ms and cpu=0.30ms after 4.84 s: they are not doing anything and never will.The detection covers java.util.concurrent locks as well. A deadlock between two ReentrantLock.lock() calls is reported in the same section, worded differently, and the threads are parked instead of blocked:
Thread dumpjcmd <pid> Thread.print -l (DeadlockTimeouts.java deadlock: the same transfers with ReentrantLock)
Real JVM output"transfer-A-to-B" #19 [655097] prio=5 os_prio=0 cpu=0.19ms elapsed=2.62s tid=0x00007d17c04b1ef0 nid=655097 waiting on condition [0x00007d17c4183000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@21.0.13/Native Method) - parking to wait for <0x00000000ff4f1b50> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) ... at java.util.concurrent.locks.ReentrantLock.lock(java.base@21.0.13/ReentrantLock.java:322) at DeadlockTimeouts.transferBlocking(DeadlockTimeouts.java:59) ... Locked ownable synchronizers: - <0x00000000ff4f1b00> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) ... Found one Java-level deadlock: ============================= "transfer-A-to-B": waiting for ownable synchronizer 0x00000000ff4f1b50, (a java.util.concurrent.locks.ReentrantLock$NonfairSync), which is held by "transfer-B-to-A" "transfer-B-to-A": waiting for ownable synchronizer 0x00000000ff4f1b00, (a java.util.concurrent.locks.ReentrantLock$NonfairSync), which is held by "transfer-A-to-B" ... Found 1 deadlock.
WAITING (parking) and parking to wait for <…b50>, a ReentrantLock$NonfairSync. With -l its Locked ownable synchronizers lists <…b00>, the lock it holds. The deadlock section says waiting for ownable synchronizer … which is held by. A search of your logs or alerts for the word BLOCKED would miss this deadlock. Search for Found one Java-level deadlock instead.Lock ordering
If every thread takes the locks in the same order, a cycle is impossible: the thread that holds the lower lock is the only one that can be waiting for the higher one. For accounts, order by account id. The transfer from B to A then locks A first, exactly as the transfer from A to B does, and one of them simply waits for the other to finish.
Program outputjava Deadlock.java ordered
Real JVM outputpid 4028, mode ordered no deadlock: A=1150 B=850 (total 2000)
tryLock, livelock and retry storms
When no global order exists, for example because the second lock is only known after the first is taken, the alternative is to stop waiting: take the first lock, try the second with tryLock, and if that fails release the first and start again. This removes hold-and-wait. It also creates a new risk. If two threads do this in step, each takes its first lock, fails on the second, releases and retries, for ever. Both are active and neither finishes. That is livelock.
Program outputjava DeadlockTimeouts.java livelock (two opposite transfers, tryLock and retry)
Real JVM outputEach transfer: lock own account, work 1 ms, tryLock(other) without waiting, on failure unlock, back off, retry. fixed 1 ms back-off run 1: A-to-B needed 1 attempts, B-to-A needed 2 attempts, both done after 8 ms (A=1150 B=850) fixed 1 ms back-off run 2: A-to-B needed 2 attempts, B-to-A needed 1 attempts, both done after 8 ms (A=1150 B=850) fixed 1 ms back-off run 3: A-to-B needed 1 attempts, B-to-A needed 2 attempts, both done after 5 ms (A=1150 B=850) random 0-4 ms back-off run 1: A-to-B needed 2 attempts, B-to-A needed 1 attempts, both done after 6 ms (A=1150 B=850) random 0-4 ms back-off run 2: A-to-B needed 2 attempts, B-to-A needed 1 attempts, both done after 6 ms (A=1150 B=850) random 0-4 ms back-off run 3: A-to-B needed 2 attempts, B-to-A needed 1 attempts, both done after 10 ms (A=1150 B=850)
Program outputjava -XX:ActiveProcessorCount=2 -Xmx64m DeadlockTimeouts.java retry
Real JVM output8 threads x 50,000 transfers between 4 accounts, 2 CPUs. Both locks are held for a few microseconds. tryLock, retry at once 9,976 ms failed attempts 6,549,736,066 process CPU 15,300 ms 40,096 transfers/s total balance 4,000,000 tryLock, random back-off 1,166 ms failed attempts 32,540 process CPU 840 ms 343,053 transfers/s total balance 4,000,000 lock() in account-id order 711 ms failed attempts 0 process CPU 680 ms 562,587 transfers/s total balance 4,000,000
tryLock and took CPU time away from the thread that held it. Random back-off: 1,166 ms, 32,540 failed attempts, 840 ms of CPU. Blocking locks in id order: 711 ms, no failed attempts, 680 ms of CPU. All three end with the correct total. No deadlock, and no thread is ever BLOCKED in the first variant: a dump shows only RUNNABLE threads in the retry loop.Starvation
A starved thread is able to run and never gets its turn. Java's monitors and the default ReentrantLock are not fair: when a lock is released, a thread that is already running can take it ahead of threads that have waited longer. Usually this evens out. It does not when some threads hold the lock for long periods and re-acquire it at once, or when one kind of work always arrives before another.
Program outputjava -XX:ActiveProcessorCount=2 -Xmx128m LocksBench.java (last part)
Real JVM outputFairness: 8 threads take one ReentrantLock in a loop for 1 s new ReentrantLock() 26,679,730 acquisitions/s fewest per thread 3,161,952 most per thread 3,791,635 new ReentrantLock(true) 1,470,684 acquisitions/s fewest per thread 132,452 most per thread 259,779
new ReentrantLock(true)) guarantees the order of arrival and did 18 times fewer acquisitions in total. Fairness is a cure for a measured starvation problem, and an expensive one.- Writer starvation on a read-write lock: a steady stream of readers keeps the read lock held, and a writer waits for a gap that never comes. See the locks lesson.
- Thread-pool starvation: every worker waits for a task that is queued behind them on the same pool. No lock is involved and no deadlock is reported. See CompletableFuture starvation.
- Priority and greedy threads: one thread calls a long
synchronizedmethod in a tight loop, and others that need the same monitor are blocked most of the time.
Lock contention
Contention is what a lock costs when it is busy. If each operation spends a fraction of its time inside one shared lock, that fraction runs one thread at a time whatever the number of threads and CPUs. If 70% of each operation is inside the lock, throughput can never exceed about 1.4 times that of a single thread. Threads beyond that only lengthen the queue. Each handover also costs: a blocked thread has to be woken and scheduled before it can use the lock.
Program outputjava LockContention.java global and java LockContention.java striped (lab lock-contention)
Real JVM outputpid 4547, mode global, 8 threads, 2 CPUs global: 2,817,166 updates in 5 s = 563,433 updates/s pid 4575, mode striped, 8 threads, 2 CPUs striped: 5,660,929 updates in 5 s = 1,132,185 updates/s
synchronized HashMap, 563,433 updates per second. With a ConcurrentHashMap of LongAdders and the extra work moved outside any shared lock, 1,132,185: twice as much.Thread dumpjcmd 4603 Thread.print (LockContention global: three of the eight workers)
Real JVM output"view-counter-0" #23 [4626] prio=5 os_prio=0 cpu=830.58ms elapsed=3.65s tid=0x00007e8754631690 nid=4626 waiting for monitor entry [0x00007e872dbfc000] java.lang.Thread.State: BLOCKED (on object monitor) at LockContention.recordGlobal(LockContention.java:23) - waiting to lock <0x00000000f3500540> (a java.util.HashMap) ... "view-counter-1" #24 [4627] prio=5 os_prio=0 cpu=798.40ms elapsed=3.65s tid=0x00007e8754632810 nid=4627 waiting for monitor entry [0x00007e872dafc000] java.lang.Thread.State: BLOCKED (on object monitor) at LockContention.recordGlobal(LockContention.java:23) - waiting to lock <0x00000000f3500540> (a java.util.HashMap) ... "view-counter-5" #28 [4631] prio=5 os_prio=0 cpu=890.45ms elapsed=3.65s tid=0x00007e87546375f0 nid=4631 runnable [0x00007e872d6fb000] java.lang.Thread.State: RUNNABLE at LockContention.spin(LockContention.java:37) at LockContention.recordGlobal(LockContention.java:25) - locked <0x00000000f3500540> (a java.util.HashMap)
view-counter threads are BLOCKED (on object monitor), all waiting to lock <0x00000000f3500540>, a java.util.HashMap. The eighth, view-counter-5, is RUNNABLE and its stack has - locked <0x00000000f3500540>: it is the owner, and it is at LockContention.spin, doing work inside the lock. There is no deadlock section. Look also at cpu=830.58ms elapsed=3.65s: this thread was on a CPU for less than a quarter of its life. Take a second dump and the owner will be a different thread: that is contention. In a deadlock the same threads hold the same locks in every dump.JFRjfr print --events jdk.JavaMonitorEnter and jfr view contention-by-site (same run, settings=profile)
Real JVM outputjdk.JavaMonitorEnter {
startTime = 21:26:35.120 (2026-11-15)
duration = 12.9 ms
monitorClass = java.util.HashMap (classLoader = bootstrap)
previousOwner = "view-counter-1" (javaThreadId = 24)
address = 0x7E86CC004D10
eventThread = "view-counter-7" (javaThreadId = 30)
stackTrace = [
LockContention.recordGlobal(String) line: 24
LockContention.lambda$main$1(CountDownLatch, long, int, String, LongAdder) line: 58
java.lang.Thread.run() line: 1583
...
]
}
Contention by Site
StackTrace Count Avg. Max.
--------------------------------------------------------- ----- ------- -------
LockContention.recordGlobal(String) 894 22.7 ms 58.7 msjdk.JavaMonitorEnter event is one wait to enter a monitor. duration = 12.9 ms is how long view-counter-7 waited, monitorClass is the class of the locked object, previousOwner is the thread that held it, and the stack gives the line that wanted the lock: LockContention.recordGlobal(String) line: 24. The view below it adds them up: 894 waits at that one site, 22.7 ms on average and 58.7 ms at worst, for an operation that takes microseconds. Only waits of at least 10 ms are recorded with settings=profile (20 ms with the default settings), so the true number of waits is higher.Architecture
Drawing diagram…
Flowchart. Take a thread dump with lock information. If it ends with a deadlock section, it is a deadlock: restart and fix the lock order. Otherwise, if many threads wait on one lock address, check the owner: an owner that changes between dumps means contention, the same owner in a slow call means a lock held across input or output, and no owner means a leaked or read lock or a non-lock wait. If few threads wait but CPU is high with threads in a retry loop, it is a livelock or retry storm. Otherwise look at pool starvation, garbage collection or a slow dependency.
flowchart TD
A["Requests hang or are slow"] --> B["Take a thread dump with jcmd Thread.print -l"]
B --> C{"Does it end with Found one Java-level deadlock"}
C -->|"Yes"| D["Deadlock. Restart, then fix the lock order at the lines shown"]
C -->|"No"| E{"Are many threads BLOCKED or parked on one address"}
E -->|"Yes"| F{"Is there an owner, and does it change between dumps"}
F -->|"Owner changes"| G["Lock contention. Shorten or split the critical section"]
F -->|"Same owner, in a slow call"| H["Lock held across I/O. Move the call out of the lock"]
F -->|"No owner listed"| I["Leaked or read lock, or a non-lock wait. Not detectable as deadlock"]
E -->|"No"| J{"Is CPU high with threads RUNNABLE in a retry loop"}
J -->|"Yes"| K["Livelock or retry storm. Add random back-off and a retry limit"]
J -->|"No"| L["Look elsewhere, such as pool starvation, GC or a slow dependency"]- The deadlock section is the first thing to check because it is unambiguous and printed by the JVM itself. Its absence rules out only cycles of owned locks.
- For everything else you need two dumps, 5 to 10 seconds apart. One dump cannot tell a thread that is stuck from a thread that happened to be there.
- Group waiting threads by the address in
waiting to lock <…>orparking to wait for <…>. One address with many waiters is one lock. Then find the thread whose stack has- locked <same address>, or which lists it underLocked ownable synchronizers. - The livelock branch has no waiting threads at all. The evidence is CPU time and the same
RUNNABLEframes in every dump, with counters that show retries rising and completions flat.
Step by step
Drawing diagram…
Sequence diagram. In the first half, the transfer from A to B locks account A while the transfer from B to A locks account B. Each then asks for the other account and blocks while holding its own, which closes a cycle. In the second half both transfers ask for account A first. The second one waits, the first takes B, finishes and unlocks, and the second continues.
sequenceDiagram participant T1 as transfer A to B participant A as Account A lock participant B as Account B lock participant T2 as transfer B to A T1->>A: lock, acquired T2->>B: lock, acquired T1->>B: lock, held by the other transfer Note over T1: BLOCKED, still holding A T2->>A: lock, held by the other transfer Note over T2: BLOCKED, still holding B Note over T1,T2: Cycle closed. Neither will ever release Note over A,B: With one global order both ask for A first T1->>A: lock, acquired T2->>A: lock, waits its turn T1->>B: lock, acquired T1->>A: unlock both A-->>T2: acquired, continues to B
- The deadlock needs both first locks to be taken before either second lock is requested. That window is a few instructions wide, which is why the bug can sit in production for months.
- After the cycle closes, nothing that happens later can open it. Timeouts at the HTTP client free the caller, not these two threads.
- With one order, the second transfer waits before it holds anything. A thread that holds nothing cannot be part of a cycle.
- The ordering rule is about the locks, not the direction of the business operation. The transfer from B to A still debits B and credits A.
In a real system
A wallet service keeps balances in memory and moves money between accounts. Once every few weeks, a handful of requests never return. The count of busy request threads climbs by two at a time and never comes down, and after some days the instance is restarted for an unrelated deploy and the problem resets. This is the usual shape of a production deadlock: small, permanent, and found late. The Investigate case Requests that never return walks through one.
- First action. Take a thread dump before restarting:
jcmd <pid> Thread.print -l > /tmp/dump-1.txt. A restart destroys the only evidence. Then restart: a deadlock does not heal. - Reading it. Go to the end of the file. If
Found one Java-level deadlockis there, the twowaiting to lock/lockedpairs underJava stack informationname the lines to fix. - Finding it earlier. A scheduled check with
ThreadMXBean.findDeadlockedThreads()can log or alert within seconds. The documentation says the method is meant for troubleshooting and might be expensive, so run it every 30 seconds, not on every request. - Preventing it. Keep the code that takes two locks in one place and order by a stable key. Do not call code you do not control, such as listeners or callbacks, while holding a lock: it may take locks of its own in an order you cannot see.
Contention is far more common. A product service counts page views in one synchronized map. At normal traffic nobody notices. During a sale, p99 latency rises from 20 ms to 400 ms while CPU is busy and the downstream systems are healthy. Adding pods helps, because each pod has its own lock; doubling the threads per pod does nothing. Two dumps show most request threads BLOCKED on one java.util.HashMap with a different owner each time, and JFR shows hundreds of jdk.JavaMonitorEnter events at one line. The Investigate case CPU pinned with flat traffic covers this pattern.
| You see | It is | Do |
|---|---|---|
Found one Java-level deadlock at the end of the dump | Deadlock | Save the dump, restart, impose one lock order at the two lines named |
Many threads BLOCKED on one address, owner RUNNABLE and different in each dump | Contention | Shorten the critical section, split the lock, or use a concurrent structure |
| Many threads waiting on one address, same owner each time, owner in a socket or JDBC call | Lock held across a slow call | Move the call outside the lock; add a timeout to the call |
CPU saturated, threads RUNNABLE in tryLock or a CAS loop, throughput near zero | Livelock or retry storm | Random back-off, a retry limit, or blocking locks in a fixed order |
| One kind of request times out while others are fast; its threads wait on a lock others hold in long loops | Starvation | Shorten the long holder; as a last resort a fair lock |
Java and Spring Boot
Two Spring Boot classes for the wallet service: a transfer that uses timed tryLock with random back-off, for situations where a lock order cannot be imposed, and a watchdog that reports deadlocks. They were compiled with javac on JDK 21 against stand-ins for the Spring and SLF4J types, not run inside a Spring application. The lock-ordering version is in the next section.
package com.example.wallet;import java.util.concurrent.ThreadLocalRandom;import java.util.concurrent.TimeUnit;import org.springframework.stereotype.Service;/** For code that cannot impose one lock order, for example when the second lock is only known after the first is held. */@Servicepublic class TimedTransferService { private static final int MAX_ATTEMPTS = 5; private static final long LOCK_WAIT_MILLIS = 50; private final AccountRepository accounts; public TimedTransferService(AccountRepository accounts) { this.accounts = accounts; } public void transfer(long fromId, long toId, long cents) throws InterruptedException { Account from = accounts.get(fromId); Account to = accounts.get(toId); for (int attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) { if (from.lock().tryLock(LOCK_WAIT_MILLIS, TimeUnit.MILLISECONDS)) { try { if (to.lock().tryLock(LOCK_WAIT_MILLIS, TimeUnit.MILLISECONDS)) { try { if (from.balanceCents() < cents) { throw new InsufficientFundsException(fromId); } from.add(-cents); to.add(cents); return; } finally { to.lock().unlock(); } } } finally { from.lock().unlock(); // gave up on the second lock: release the first as well } } // Wait a random time before trying again, so two transfers that collided do not collide again in step. TimeUnit.MILLISECONDS.sleep(ThreadLocalRandom.current().nextLong(5, 25L * attempt)); } throw new TransferBusyException("could not lock accounts " + fromId + " and " + toId + " after " + MAX_ATTEMPTS + " attempts"); }}Why it is written this way
- Both acquisitions have a time limit, so no thread can wait for ever. That removes hold-and-wait: a thread that cannot get the second lock gives the first one back.
- Each
unlock()is in afinallythat belongs to thetryLockthat succeeded. The outerfinallyruns whether or not the innertryLockgot the lock. - The pause is random and grows with the attempt number. A fixed pause, or none, lets colliding threads collide again; in the measurement above, retrying at once cost 6.5 billion failed attempts.
MAX_ATTEMPTSturns an unlucky transfer into an error the caller can see and retry, instead of a thread that spins without limit. A controller would mapTransferBusyExceptionto 503.Accountholds an id, a balance and aReentrantLock;AccountRepositoryreturns the oneAccountobject per id. Both are assumed from the rest of the project.
package com.example.wallet;import java.lang.management.ManagementFactory;import java.lang.management.ThreadInfo;import java.lang.management.ThreadMXBean;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.scheduling.annotation.Scheduled;import org.springframework.stereotype.Component;@Componentpublic class DeadlockWatchdog { private static final Logger log = LoggerFactory.getLogger(DeadlockWatchdog.class); private final ThreadMXBean threads = ManagementFactory.getThreadMXBean(); @Scheduled(fixedDelay = 30_000) public void check() { long[] ids = threads.findDeadlockedThreads(); // monitors and ownable synchronizers; null when there is no cycle if (ids == null) { return; } for (ThreadInfo info : threads.getThreadInfo(ids, true, true)) { log.error("DEADLOCK thread=\"{}\" state={} waitingFor={} heldBy=\"{}\"", info.getThreadName(), info.getThreadState(), info.getLockName(), info.getLockOwnerName()); } }}Why it is written this way
findDeadlockedThreads()covers monitors and ownable synchronizers such asReentrantLock.findMonitorDeadlockedThreads()would miss the second kind.- It returns
null, not an empty array, when there is no deadlock. getThreadInfo(ids, true, true)includes locked monitors and synchronizers, which gives the same facts as the deadlock section of a thread dump: thread, lock waited for, and owner.- It logs on every run while the deadlock exists. That is deliberate: the condition is permanent, and an alert on this log line should stay red until the instance is restarted.
@Scheduledneeds@EnableSchedulingon a configuration class. The check does not see cycles that include virtual threads.
Wrong, then right
The transfer method of the wallet service. The incorrect version locks the two accounts in the order of its arguments, which is the natural way to write it and is correct for any single transfer.
Incorrect implementation
package com.example.wallet;import org.springframework.stereotype.Service;@Servicepublic class TransferService { private final AccountRepository accounts; // in-memory ledger: one Account object per account id public TransferService(AccountRepository accounts) { this.accounts = accounts; } public void transfer(long fromId, long toId, long cents) { Account from = accounts.get(fromId); Account to = accounts.get(toId); synchronized (from) { // lock order = argument order synchronized (to) { if (from.balanceCents() < cents) { throw new InsufficientFundsException(fromId); } from.add(-cents); to.add(cents); } } }}Why it is written this way
- The lock order depends on the caller's arguments.
transfer(1, 2, …)locks account 1 then 2;transfer(2, 1, …)locks 2 then 1. - While waiting for
to, the thread keepsfromlocked.synchronizedhas no timeout and cannot be interrupted, so the wait has no end.
Why it fails
- Two requests move money between the same two accounts in opposite directions at the same moment. Each thread enters its outer
synchronized, then blocks on the inner one, which the other thread holds. All four deadlock conditions hold: exclusive locks, hold and wait, no preemption, and a cycle of two. - Neither thread ever returns. In the real run the dump shows both
BLOCKED (on object monitor)at the innersynchronized, with about 0.3 ms of CPU time each after 4.84 s, and the JVM printsFound one Java-level deadlock. - The damage spreads. Both accounts stay locked, so every later transfer or balance read that touches either account blocks behind them. Request threads are lost one by one, the callers see timeouts with no server-side error, and CPU stays idle so autoscaling does nothing.
- It cannot be reproduced on demand. A single-threaded test passes, and a load test hits it only if two opposite transfers on one pair of accounts overlap within microseconds.
Corrected implementation
package com.example.wallet;import org.springframework.stereotype.Service;@Servicepublic class TransferService { private final AccountRepository accounts; // in-memory ledger: one Account object per account id public TransferService(AccountRepository accounts) { this.accounts = accounts; } public void transfer(long fromId, long toId, long cents) { if (fromId == toId) { throw new IllegalArgumentException("source and target are the same account"); } Account from = accounts.get(fromId); Account to = accounts.get(toId); // One global order for every thread: the account with the smaller id is always locked first, // whichever direction the money moves. Account first = from.id() < to.id() ? from : to; Account second = first == from ? to : from; synchronized (first) { synchronized (second) { if (from.balanceCents() < cents) { throw new InsufficientFundsException(fromId); } from.add(-cents); to.add(cents); } } }}Why it is written this way
firstis always the account with the smaller id. Two opposite transfers now both ask for the same account first, so one waits before it holds anything.- The business direction is unchanged: money still leaves
fromand arrives atto. Only the order of locking is fixed. - The same-account check is needed for correctness of the rule: with equal ids there is no order, and the transfer would be meaningless anyway.
- The id must be stable and unique. Ordering by
System.identityHashCodeis a fallback when there is no key, and needs a tie-breaking lock for the rare equal hash.
Trade-offs
- The rule is global and nothing enforces it. Every method that locks two accounts must use the same order, and one new method that locks in argument order brings the deadlock back. Keep all two-account locking in one helper.
- It covers only these locks. If code inside the block takes another lock, calls a listener, or waits for a database row lock, a cycle can still form through that resource, and the JVM will not see the database part of it.
- A transfer between two busy accounts now waits its turn with no time limit. That is correct, but a slow holder still delays everyone behind it; ordering prevents deadlock, not contention.
- In-memory locks protect one JVM. With several instances of the service, the same two accounts can be changed on two machines at once, and the ordering has to move to where the data is, for example row locks taken in id order inside one database transaction.
Configuration
Deadlocks have no setting. What can be configured is what the diagnostic tools show. Defaults are those of JDK 21, read from the tool reference and from lib/jfr/default.jfc and profile.jfc in the JDK.
jcmd <pid> Thread.print -lDefault -l is off
- What it does
- Adds
Locked ownable synchronizersto each thread: theReentrantLocks and write locks it holds. Monitors held are always shown as- locked <…>in the stack. - When to change it
- Use
-lwhenever threads areparking to wait fora lock. The deadlock section is printed with or without it.
jdk.JavaMonitorEnter#threshold (JFR)Default 20 ms in default.jfc, 10 ms in profile.jfc
- What it does
- Waits to enter a
synchronizedblock shorter than this are not recorded. - When to change it
- For a short investigation lower it, for example
jcmd <pid> JFR.start settings=profile jdk.JavaMonitorEnter#threshold=1ms duration=60s filename=/tmp/locks.jfr. Many short waits can cost more in total than a few long ones.
jdk.ThreadPark#threshold (JFR)Default 20 ms in default.jfc, 10 ms in profile.jfc
- What it does
- The matching event for waits on
java.util.concurrentlocks, which do not producejdk.JavaMonitorEnter. - When to change it
- Include it when the contended lock is a
ReentrantLock.jfr view contention-by-sitecounts monitor waits only.
new ReentrantLock(boolean fair)Default false (non-fair)
- What it does
- A fair lock is granted in order of arrival, which prevents starvation of waiting threads.
- When to change it
- Turn it on only for a measured starvation problem. In this lesson's run a fair lock did 18 times fewer acquisitions per second.
Hands-on exercise
Build a deadlock, read it, and remove one of its conditions
Step two opposite transfers into a deadlock, read the wait-for cycle and the simulated dump, then prevent it first with a global lock order and then with a tryLock timeout.
Steps · 0 of 6 done
Two transfers, two locks
Simulation · not a real JVMA simulation of the lab program. You decide which thread takes the next step, so you can produce the unlucky interleaving on purpose, then change one rule and try again.
What you should see
With opposite ordering, once each transfer holds its first lock the next step blocks both: each thread holds one account and waits for the other, the graph is a cycle of two, and no further step is possible. With the global order, the same steps end differently: the second transfer waits for the first lock while holding nothing, the first transfer takes both locks and finishes, and then the second runs. Both complete with A=1150 and B=850, as in the real run of Deadlock.java ordered. With the tryLock timeout, the cycle forms but a timeout makes one thread release its lock, the other finishes, and the first retries: the transfers complete at the cost of at least one timeout and retry.
What can go wrong
Lock order taken from the arguments
- Symptom
- Rarely, two requests never return and the threads serving them stay
BLOCKEDfor ever. The dump ends withFound one Java-level deadlock. - Cause
- Two code paths, or one path called with swapped arguments, take the same two locks in opposite orders.
- Fix
- Order the locks by a stable unique key before taking them, in one shared helper. Review every place that holds one of these locks while taking another.
Calling foreign code while holding a lock
- Symptom
- A deadlock whose second lock is in a library, a listener, a logger appender or a cache loader that your code never mentions.
- Cause
- A callback invoked inside a
synchronizedblock takes its own lock, while another thread holds that lock and calls back into your class. - Fix
- Copy what the callback needs while holding the lock, release the lock, then call out. Keep critical sections free of calls you do not control.
tryLock retry without back-off
- Symptom
- CPU at 100%, throughput close to zero, no
BLOCKEDthreads, no deadlock section. Dumps show the same threadsRUNNABLEin a loop aroundtryLock. - Cause
- Failed attempts retry at once. Spinning threads take CPU time from the thread that holds the lock, so the lock is held longer and more attempts fail.
- Fix
- Wait a random time between attempts, limit the attempts, and fail visibly after the limit. Where a lock order exists, prefer it: it needs no retries at all.
Reading many BLOCKED threads as a deadlock
- Symptom
- A team reorders locks or restarts the service after seeing 150
BLOCKEDthreads. The slowness returns under the next load peak. - Cause
- The threads were queueing behind an owner that was doing work. There was no cycle, so there was no deadlock to fix.
- Fix
- Check for the deadlock section first. Without it, find the owner and what it does while holding the lock, and compare two dumps a few seconds apart.
Adding threads to a contended lock
- Symptom
- The pool is raised from 50 to 200 threads. Throughput does not improve, latency gets worse, and more threads show as
BLOCKED. - Cause
- The part of each request inside the lock runs one thread at a time. More threads only add waiters and context switches.
- Fix
- Reduce the time inside the lock, or the sharing: move work out of the critical section, use
ConcurrentHashMapandLongAdder, or split one lock into several.
Restarting before taking a dump
- Symptom
- The hang comes back every few weeks and nobody knows why. Each time the instance was restarted to restore service.
- Cause
- The thread dump is the only record of which threads and locks formed the cycle, and a restart erases it.
- Fix
- Make
jcmd <pid> Thread.print -lto a file part of the restart procedure, or run a watchdog that logsfindDeadlockedThreads().
How to troubleshoot
Some requests never return and CPU is idle
Check
jcmd <pid> Thread.print -l > /tmp/dump-1.txt then grep -n -A12 'Found one Java-level deadlock' /tmp/dump-1.txt
- How to read it
- If the section exists, it lists each thread, the lock it waits for and the thread holding it. The stacks below it show the two lines with
waiting to lockandlocked. Those lines take the locks in opposite orders.
Service is slow under load and many threads are BLOCKED
Check
grep -c 'java.lang.Thread.State: BLOCKED' /tmp/dump-1.txt and grep 'waiting to lock' /tmp/dump-1.txt | sort | uniq -c | sort -rn | head
- How to read it
- The first number is how many threads wait for monitors. The second groups them by lock. One address with most of the waiters is the contended lock. In the lab dump it is 7 threads on one
java.util.HashMap.
Need the owner of the contended monitor
Check
grep -n -B12 -e '- locked <0x00000000f3500540>' /tmp/dump-1.txt (use the address from the previous step)
- How to read it
- The thread whose stack contains
- locked <that address>owns it.RUNNABLEin application code means work inside the lock. A socket or database frame means a slow call inside the lock. If the owner is itselfBLOCKEDon another lock, follow the chain.
Need to know which lock costs the most time
Check
jcmd <pid> JFR.start settings=profile duration=60s filename=/tmp/locks.jfr then jfr view contention-by-site /tmp/locks.jfr and jfr print --events jdk.JavaMonitorEnter /tmp/locks.jfr
- How to read it
- The view lists each call site with the number of waits and their average and maximum duration. Count times average is the time lost at that site. In the lab run: 894 waits averaging 22.7 ms at
LockContention.recordGlobal.
CPU is saturated, throughput is near zero, nothing is BLOCKED
Check
Take three dumps a second apart with jcmd <pid> Thread.print and compare the top frames of the RUNNABLE application threads; top -H -p <pid> shows which threads use the CPU
- How to read it
- The same threads in the same retry loop (
tryLock,compareAndSet) in every dump, all using CPU, is a livelock or retry storm. Threads that areRUNNABLEin different places each time are doing real work.
Want to detect a deadlock from inside the application
Check
ManagementFactory.getThreadMXBean().findDeadlockedThreads() (null means none)
- How to read it
- A non-null array holds the ids of the threads in the cycle. Pass it to
getThreadInfo(ids, true, true)for names, the lock each waits for and its owner. Cycles that include virtual threads are not found.
Key takeaways
- A deadlock needs exclusive locks, hold and wait, no preemption and a cycle. One global lock order removes the cycle and costs nothing at run time.
- The JVM finds deadlocks for you: the dump ends with
Found one Java-level deadlock, and thewaiting to lockandlockedpairs name the lines. It coversReentrantLocktoo, asownable synchronizer. - A deadlock never heals. Save a dump, then restart.
- Many
BLOCKEDthreads without that section is contention. Find the owner of the address they wait for and look at what it does inside the lock. tryLockwith a timeout removes hold-and-wait but needs random back-off and a retry limit. Retrying at once burned 15 s of CPU for work that needed 0.7 s.- Contention makes part of every request run one thread at a time. More threads lengthen the queue; shrinking or splitting the lock doubled throughput in the lab.
- JFR measures it:
jdk.JavaMonitorEnterevents andjfr view contention-by-sitefor monitors,jdk.ThreadParkforjava.util.concurrentlocks.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A pricing service is slow. A dump shows 180 of 200 request threads BLOCKED, all 'waiting to lock <0x00000007a1c0e2f8> (a com.example.PriceCache)'. One thread is RUNNABLE in SocketInputStream.read with '- locked <0x00000007a1c0e2f8>' further down its stack. The dump has no deadlock section. A second dump 10 s later shows the same owner in the same frame.
What is the problem and the fitting fix?
Scenario 2 of 3
An order service hangs partially every few weeks. A dump taken during the hang ends with 'Found one Java-level deadlock': thread http-17 is at OrderService.reserve and waits for an Inventory monitor while it has locked an Order; thread http-44 is at InventoryService.restock and waits for that Order while it has locked the Inventory.
What should the team do?
Scenario 3 of 3
To avoid deadlocks a team replaced nested synchronized blocks with a loop: lock the first account, tryLock() the second, and on failure unlock and retry at once. In a load test with 64 threads, CPU is at 100% on all cores, completed transfers per second fall to a tenth, no thread is BLOCKED, and the dump has no deadlock section.
What explains the result and what should change?
References
References: Java SE 21 API: ThreadMXBean (findDeadlockedThreads, monitors and ownable synchronizers, virtual threads not covered) ↗ · JDK 21 tool reference: jcmd (Thread.print and its -l option, JFR.start) ↗ · JDK 21 tool reference: jfr (print, summary, view) ↗ · Java SE 21 API: Thread.State (BLOCKED, WAITING, TIMED_WAITING) ↗ · Java SE 21 API: ReentrantLock (tryLock with timeout, fairness and its throughput cost) ↗ · The Java Tutorials: Deadlock ↗ · The Java Tutorials: Starvation and Livelock ↗ · Report a problem