Skip to content
Intermediate20 minSimulation 3 · Java 21

Deadlock: two transfers, two locks, opposite order

Step two money transfers into a deadlock, read the wait-for graph and the thread dump that names it, then fix it with one global lock order or a tryLock timeout.

The situation

A Spring Boot 3 ledger service moves money between in-memory account objects. Each transfer locks the source account, then the target. It ran for weeks without a problem. Then two customers pay each other at the same moment: one request moves 100 from account A to B, the other moves 250 from B to A. Both requests hang until the client gives up after 30 seconds. So does every later request that touches A or B. CPU usage is flat, the heap is quiet and the log has no error. Each such pair takes 2 of Tomcat's 200 worker threads for good, and only a restart gives them back.

Read first: Deadlocks, livelocks, starvation and lock contention, Locks: ReentrantLock, ReadWriteLock and StampedLock

Break it in the simulation

Step two transfers into a deadlock

Simulation · not a real JVM

Let each thread take its first lock, then step either one. Watch the wait-for graph close into a cycle and read the simulated thread dump. Then switch the lock ordering to Global order and repeat the same steps.

Guided run · 0 of 5 stages done

    1. Leave Lock ordering on Opposite order and tryLock with timeout off.
    2. Press the button of transfer-A-to-B three times: lock A, lock B, move money and unlock.
    3. Then press the button of transfer-B-to-A three times.

    Look for

    • Account A shows 1150 and account B shows 850. Total across both accounts is 2000.
    • The verdict says both transfers finished, by luck: one thread had both locks before the other took its first. This is the timing every test and almost every production request gets.
    1. Press Reset.
    2. Press lock A on transfer-A-to-B, then lock B on transfer-B-to-A. Each thread now holds one lock.
    3. Press lock B on transfer-A-to-B. Then press lock A on transfer-B-to-A.
    4. Or press Run to deadlock, which does the same four steps.

    Look for

    • After the third step transfer-A-to-B is BLOCKED (on object monitor) and waits for B. There is no cycle yet, but the owner of B still needs A, so its next step will close it.
    • After the fourth step both threads are BLOCKED and both buttons are disabled: BLOCKED on B and BLOCKED on A.
    • Both balances are still 1000. No money moved, nothing failed, and nothing will ever change again.
    1. Follow the arrows in Wait-for graph, starting at transfer-A-to-B.
    2. Read the panel Thread dump, deadlock section.
    3. Compare it with the real dump under the lab on this page.

    Look for

    • transfer-A-to-B waits for lock B, which is held by transfer-B-to-A, which waits for lock A, which is held by transfer-A-to-B. You are back where you started: a cycle.
    • The dump begins with Found one Java-level deadlock: and says for each thread which monitor it is waiting to lock and which thread it is held by.
    • In the stack part, each thread has one - waiting to lock <address> line and one - locked <address> line, and the two addresses are swapped between the threads. That swap is the opposite lock order, visible in the dump.
    1. Set Lock ordering to Global order.
    2. Check Lock order on both thread cards.
    3. Press Run to deadlock: the same alternating timing as before.

    Look for

    • Both threads now take A then B, whichever way the money moves.
    • In Steps taken, step 2: transfer-B-to-A asks for A, which is taken, and is BLOCKED. It holds nothing while it waits, so no cycle can form.
    • The run finishes after 7 steps with A=1150 and B=850, the balances the real lab printed.
    • Press Reset and try by hand to produce a cycle. You cannot: the thread that loses the race for A never holds B.
    1. Set Lock ordering back to Opposite order and switch tryLock with timeout on.
    2. Press Run to deadlock.
    3. Press Reset. Press tryLock A on the first thread, tryLock B on the second, then tryLock B on the first and tryLock A on the second. Now press both timeout: unlock …, retry buttons, and repeat the same six steps.

    Look for

    • Run to deadlock now finishes: A=1150, B=850, and tryLock timeouts is 1. Step 5 in Steps taken is the timeout: transfer-A-to-B unlocks A, and that lets transfer-B-to-A through.
    • While both threads wait, the graph shows the same cycle and the verdict says "for now". The states are TIMED_WAITING, not BLOCKED: each wait has an end.
    • Stepping by hand in the same rhythm, both threads time out, both retry and both collide again, for as long as you keep going. Nobody is blocked and nothing gets done: a livelock. Real code waits a random time before retrying to break the rhythm.

Wrong, then right

Incorrect implementation

src/main/java/com/example/ledger/TransferService.javaJava
package com.example.ledger;import java.math.BigDecimal;import java.util.Map;import java.util.concurrent.ConcurrentHashMap;import org.springframework.stereotype.Service;@Servicepublic class TransferService {    static final class Account {        final long id;        BigDecimal balance;        Account(long id, BigDecimal balance) {            this.id = id;            this.balance = balance;        }    }    private final Map<Long, Account> accounts = new ConcurrentHashMap<>();    /** Called from request threads. Two opposite transfers between the same accounts can deadlock. */    public void transfer(long fromId, long toId, BigDecimal amount) {        Account from = accounts.get(fromId);        Account to = accounts.get(toId);        synchronized (from) {          // thread 1 holds A, thread 2 holds B            synchronized (to) {        // thread 1 wants B, thread 2 wants A                if (from.balance.compareTo(amount) < 0) {                    throw new IllegalStateException("insufficient funds");                }                from.balance = from.balance.subtract(amount);                to.balance = to.balance.add(amount);            }        }    }}

Why it is written this way

  • The lock order follows the arguments. transfer(1, 2, …) locks account 1 then 2; transfer(2, 1, …) locks 2 then 1. The order is decided by the caller, which means by the customer.
  • Each lock on its own is correct, and so is each transfer on its own. The bug exists only in the combination of two calls.
  • synchronized has no timeout and cannot be interrupted while it waits for the monitor. Once two threads are in the cycle, nothing inside the JVM will end it.

Why it fails

  • A deadlock needs four things at once: locks that only one thread can hold, a thread that holds one lock while it waits for another, no way to take a lock away from its owner, and a cycle of threads each waiting for the next. synchronized gives the first three for free. The argument-dependent lock order supplies the cycle.
  • The window is the time between synchronized (from) and synchronized (to). If a transfer from A to B and one from B to A are both inside that window, the first holds A and wants B and the second holds B and wants A. Neither can go forward and neither will go back.
  • The window is tiny, so the bug passes tests and can run for weeks. The lab makes it certain with a latch that holds both threads between the two locks.
  • The JVM does not resolve a deadlock. The two threads stay BLOCKED and use no CPU, so no CPU, memory or GC alert fires. Every later request that needs either account blocks behind them and takes another worker thread, until the pool is empty and the whole service stops answering.

Corrected implementation

src/main/java/com/example/ledger/TransferService.javaJava
package com.example.ledger;import java.math.BigDecimal;import java.util.Map;import java.util.concurrent.ConcurrentHashMap;import java.util.concurrent.TimeUnit;import java.util.concurrent.locks.ReentrantLock;import org.springframework.stereotype.Service;@Servicepublic class TransferService {    static final class Account {        final long id;        final ReentrantLock lock = new ReentrantLock();        BigDecimal balance;        Account(long id, BigDecimal balance) {            this.id = id;            this.balance = balance;        }    }    private final Map<Long, Account> accounts = new ConcurrentHashMap<>();    /** Fix 1: one global lock order. The account with the smaller id is always locked first. */    public void transfer(long fromId, long toId, BigDecimal amount) {        if (fromId == toId) {            throw new IllegalArgumentException("same account");        }        Account from = accounts.get(fromId);        Account to = accounts.get(toId);        Account first = from.id < to.id ? from : to;        Account second = first == from ? to : from;        synchronized (first) {            synchronized (second) {                move(from, to, amount);            }        }    }    /** Fix 2, when no global order exists: give up after a timeout instead of waiting forever. */    public boolean tryTransfer(long fromId, long toId, BigDecimal amount) throws InterruptedException {        Account from = accounts.get(fromId);        Account to = accounts.get(toId);        if (!from.lock.tryLock(50, TimeUnit.MILLISECONDS)) {            return false;        }        try {            if (!to.lock.tryLock(50, TimeUnit.MILLISECONDS)) {                return false;              // the finally below releases "from"; the caller retries later            }            try {                move(from, to, amount);                return true;            } finally {                to.lock.unlock();            }        } finally {            from.lock.unlock();        }    }    private static void move(Account from, Account to, BigDecimal amount) {        if (from.balance.compareTo(amount) < 0) {            throw new IllegalStateException("insufficient funds");        }        from.balance = from.balance.subtract(amount);        to.balance = to.balance.add(amount);    }}

Why it is written this way

  • transfer: the order no longer depends on the direction. Both opposite transfers lock the account with the smaller id first, so the one that loses the race waits while holding nothing.
  • The order must be total and stable: a unique id works. System.identityHashCode can collide and needs a tie-break lock; avoid it when a real id exists.
  • fromId == toId is rejected early. With synchronized a thread can re-enter its own monitor, so it would not deadlock, but a transfer to the same account is a caller bug.
  • tryTransfer: for cases where no global order exists, for example locks owned by different libraries. tryLock with a timeout returns false instead of waiting forever, and the finally blocks release whatever was taken.
  • Every tryLock that returned true is paired with unlock() in a finally. A ReentrantLock is not released when the block is left, unlike synchronized.
  • Not compiled against Spring here: the class was compiled with JDK 21 using a stub for @Service.

Trade-offs

  • Lock ordering costs nothing at run time, but it is a rule every piece of code that takes two of these locks has to follow. One method that locks in argument order brings the deadlock back. Keep the two-lock code in one place.
  • Ordering needs all the locks to be known up front. If the second lock is only discovered while holding the first (a callback, a listener, code from another team), you cannot sort them.
  • tryLock with a timeout turns a permanent hang into a slow or failed request. The caller now has to handle false: retry, or fail the request. Retrying needs a limit and a random pause, or two threads can keep colliding (livelock).
  • A timed tryLock adds latency under contention: a request that would have waited 2 ms for a busy lock can instead time out at 50 ms and start again.
  • A single lock for all transfers cannot deadlock either and is the simplest fix, but then only one transfer runs at a time. See lock contention for what that costs.
  • All of this covers one JVM. If the balances live in a database, the locks that matter are row locks, and the same rule applies there: update rows in a fixed order.

Run it on a real JVM

Deadlock.java starts two threads, transfer-A-to-B and transfer-B-to-A, on two accounts with 1,000 each. In broken mode each locks its source account first, and a latch holds both between the two locks so that the deadlock happens on every run. The program then asks ThreadMXBean.findDeadlockedThreads() and stays hung, so you can take a thread dump. In ordered mode both lock the smaller account id first and the program prints the final balances. It proves that the same two transfers hang or finish depending only on the lock order.

With Docker (CPU and memory limits)

labs/jvm/deadlockShell
cd labs/jvm/deadlockdocker build -t jvm-lab-deadlock .# broken: prints the deadlocked threads after about 4 seconds, then hangsdocker run --rm --name deadlock-lab --cpus=2 --memory=512m jvm-lab-deadlock Deadlock broken# second terminal: thread dump of the hung JVM (java is process 1 in the container)docker exec deadlock-lab jcmd 1 Thread.print# only the deadlock report at the end of the dumpdocker exec deadlock-lab jcmd 1 Thread.print | tail -n 35# the JVM will never exit by itself: stop the containerdocker stop deadlock-lab# fixed: one global lock orderdocker run --rm --cpus=2 --memory=512m jvm-lab-deadlock Deadlock ordered

Without Docker, on a local JDK 21

labs/jvm/deadlockShell
cd labs/jvm/deadlockjava -XX:ActiveProcessorCount=2 Deadlock.java broken# second terminal: the program printed its pid on the first linejcmd <pid> Thread.print# back in the first terminal: Ctrl+C to end the hung JVM, then run the fixjava -XX:ActiveProcessorCount=2 Deadlock.java ordered
The program: Deadlock.java
labs/jvm/deadlock/Deadlock.javaJava
import java.lang.management.ManagementFactory;import java.lang.management.ThreadInfo;import java.lang.management.ThreadMXBean;import java.util.concurrent.CountDownLatch;/** * Lab 3: deadlock detection and resolution. * *   java Deadlock.java broken    -> two transfers take the same two locks in opposite order and hang forever *   java Deadlock.java ordered   -> both transfers take the locks in one global order and finish * * While "broken" hangs, take a thread dump from another terminal:  jcmd <pid> Thread.print */public class Deadlock {    static final class Account {        final long id;        long balance;        Account(long id, long balance) { this.id = id; this.balance = balance; }    }    /** Locks the source first, then the target. Two opposite transfers can each hold one lock and wait for the other. */    static void transferBroken(Account from, Account to, long amount, CountDownLatch bothHoldFirstLock) throws InterruptedException {        synchronized (from) {            bothHoldFirstLock.countDown();            bothHoldFirstLock.await(); // make the unlucky timing certain, so the lab is repeatable            synchronized (to) {                from.balance -= amount;                to.balance += amount;            }        }    }    /** Always locks the account with the smaller id first, whichever direction the money moves. */    static void transferOrdered(Account from, Account to, long amount) {        Account first = from.id < to.id ? from : to;        Account second = first == from ? to : from;        synchronized (first) {            synchronized (second) {                from.balance -= amount;                to.balance += amount;            }        }    }    public static void main(String[] args) throws Exception {        boolean broken = args.length == 0 || args[0].equals("broken");        Account a = new Account(1, 1_000);        Account b = new Account(2, 1_000);        CountDownLatch latch = new CountDownLatch(2);        Thread t1 = new Thread(() -> run(broken, a, b, 100, latch), "transfer-A-to-B");        Thread t2 = new Thread(() -> run(broken, b, a, 250, latch), "transfer-B-to-A");        t1.start();        t2.start();        System.out.println("pid " + ProcessHandle.current().pid() + ", mode " + (broken ? "broken" : "ordered"));        t1.join(2_000);        t2.join(2_000);        ThreadMXBean mx = ManagementFactory.getThreadMXBean();        long[] ids = mx.findDeadlockedThreads();        if (ids == null) {            System.out.printf("no deadlock: A=%d B=%d (total %d)%n", a.balance, b.balance, a.balance + b.balance);            return;        }        System.out.println("ThreadMXBean.findDeadlockedThreads() found " + ids.length + " threads:");        for (ThreadInfo info : mx.getThreadInfo(ids, true, true)) {            System.out.printf("  \"%s\" %s, waiting for %s held by \"%s\"%n", info.getThreadName(), info.getThreadState(), info.getLockName(), info.getLockOwnerName());        }        System.out.println("The process will now hang. Take a thread dump with: jcmd " + ProcessHandle.current().pid() + " Thread.print");    }    static void run(boolean broken, Account from, Account to, long amount, CountDownLatch latch) {        try {            if (broken) transferBroken(from, to, amount, latch);            else transferOrdered(from, to, amount);        } catch (InterruptedException e) {            Thread.currentThread().interrupt();        }    }}
The Dockerfile
labs/jvm/deadlock/DockerfileDockerfile
# Java 21 lab image. Build:  docker build -t jvm-lab-deadlock .FROM eclipse-temurin:21-jdkWORKDIR /labCOPY Deadlock.java .RUN javac -d out Deadlock.java# The JVM reads the container's CPU and memory limits (container support is on by default),# so --cpus and --memory on "docker run" change what Runtime.availableProcessors() and the default heap size report.ENTRYPOINT ["java", "-cp", "out"]CMD ["Deadlock"]

$ java -XX:ActiveProcessorCount=2 Deadlock.java broken

Program outputThe program finds its own deadlock

Real JVM output
pid 3961, mode broken
ThreadMXBean.findDeadlockedThreads() found 2 threads:
  "transfer-A-to-B" BLOCKED, waiting for Deadlock$Account@71d44a3 held by "transfer-B-to-A"
  "transfer-B-to-A" BLOCKED, waiting for Deadlock$Account@14cd1699 held by "transfer-A-to-B"
The process will now hang. Take a thread dump with: jcmd 3961 Thread.print
ThreadMXBean.findDeadlockedThreads() returned 2 threads. Each line gives the thread, its state (BLOCKED), the lock it waits for and the thread that holds it. Read the two lines together: A-to-B waits for a lock held by B-to-A, and B-to-A waits for a lock held by A-to-B. A service can run this check itself and raise an alert.

$ jcmd 3961 Thread.print

Thread dumpThread dump: the two blocked threads

Real JVM output
"transfer-A-to-B" #19 [3981] prio=5 os_prio=0 cpu=0.26ms elapsed=4.84s tid=0x00007cff8863bb70 nid=3981 waiting for monitor entry  [0x00007cff8c732000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at Deadlock.transferBroken(Deadlock.java:27)
	- waiting to lock <0x00000000f7654bc0> (a Deadlock$Account)
	- locked <0x00000000f7654ba0> (a Deadlock$Account)
	at Deadlock.run(Deadlock.java:75)
	at Deadlock.lambda$main$0(Deadlock.java:51)
	at Deadlock$Lambda/0x00007cff14154410.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)

"transfer-B-to-A" #20 [3982] prio=5 os_prio=0 cpu=0.30ms elapsed=4.84s tid=0x00007cff8863cbf0 nid=3982 waiting for monitor entry  [0x00007cff8c632000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at Deadlock.transferBroken(Deadlock.java:27)
	- waiting to lock <0x00000000f7654ba0> (a Deadlock$Account)
	- locked <0x00000000f7654bc0> (a Deadlock$Account)
	at Deadlock.run(Deadlock.java:75)
	at Deadlock.lambda$main$1(Deadlock.java:52)
	at Deadlock$Lambda/0x00007cff14156800.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)
Both threads are BLOCKED (on object monitor) in the same frame, Deadlock.transferBroken(Deadlock.java:27), the inner synchronized. Compare the addresses: transfer-A-to-B is waiting to lock <0x…4bc0> and has locked <0x…4ba0>; transfer-B-to-A has them the other way round. Also look at cpu=0.26ms after elapsed=4.84s: the threads are not working, they are stuck.

$ jcmd 3961 Thread.print

Thread dumpThread dump: the deadlock report at the end

Real JVM output
Found one Java-level deadlock:
=============================
"transfer-A-to-B":
  waiting to lock monitor 0x00007cfef0001760 (object 0x00000000f7654bc0, a Deadlock$Account),
  which is held by "transfer-B-to-A"

"transfer-B-to-A":
  waiting to lock monitor 0x00007cfefc0015d0 (object 0x00000000f7654ba0, a Deadlock$Account),
  which is held by "transfer-A-to-B"

Java stack information for the threads listed above:
===================================================
"transfer-A-to-B":
	at Deadlock.transferBroken(Deadlock.java:27)
	- waiting to lock <0x00000000f7654bc0> (a Deadlock$Account)
	- locked <0x00000000f7654ba0> (a Deadlock$Account)
	at Deadlock.run(Deadlock.java:75)
	at Deadlock.lambda$main$0(Deadlock.java:51)
	at Deadlock$Lambda/0x00007cff14154410.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)
"transfer-B-to-A":
	at Deadlock.transferBroken(Deadlock.java:27)
	- waiting to lock <0x00000000f7654ba0> (a Deadlock$Account)
	- locked <0x00000000f7654bc0> (a Deadlock$Account)
	at Deadlock.run(Deadlock.java:75)
	at Deadlock.lambda$main$1(Deadlock.java:52)
	at Deadlock$Lambda/0x00007cff14156800.run(Unknown Source)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)

Found 1 deadlock.
The dump ends with this section because HotSpot found a cycle of threads waiting for locks. Search a dump for Found one Java-level deadlock first. The upper part is the wait-for graph in words (waiting to lock monitor … which is held by …); the lower part repeats the stacks of only the threads in the cycle. The line to fix is the first frame of each stack.

$ java -XX:ActiveProcessorCount=2 Deadlock.java ordered

Program outputWith one global lock order

Real JVM output
pid 4028, mode ordered
no deadlock: A=1150 B=850 (total 2000)
Same two transfers, same amounts. Both finish: A = 1000 − 100 + 250 = 1150, B = 1000 + 100 − 250 = 850, total still 2000. The only change is which account is locked first.

Real run and simulation: what each one shows

  • The captured files come from OpenJDK 21.0.13 run directly on Linux x86-64 with -XX:ActiveProcessorCount=2, not in Docker. The Docker commands above were not executed when this was written.
  • The real dump shows what the widget leaves out: 13 Java threads plus the VM and GC threads, of which only two matter, real addresses and line numbers, and per-thread cpu= and elapsed= values that tell a stuck thread from a busy one.
  • The real lab forces the timing with a latch, so it deadlocks on every run. The simulation lets you find the timing yourself and see that most interleavings of the broken code finish.
  • The lab has no tryLock variant. What the widget shows for it is the model only. Checked separately on the same JDK 21.0.13: a thread waiting in ReentrantLock.tryLock(timeout) is reported as TIMED_WAITING (parking), and a deadlock between two ReentrantLock.lock() calls is also reported under Found one Java-level deadlock, worded waiting for ownable synchronizer instead of waiting to lock monitor.

How you meet this in production

  • The first symptom is requests that never return, for some accounts only, while CPU is low and the heap is flat. Latency graphs show timeouts, not slowness. Over hours the number of busy worker threads climbs and never falls, and at the end the service stops answering health checks. Requests that never return walks through this case.
  • The tool is a thread dump: jcmd <pid> Thread.print. Go to the end of the output and look for Found one Java-level deadlock. It names the threads, the locks and the frames. One dump is enough: a deadlock does not change between dumps.
  • In a Spring Boot service the threaddump Actuator endpoint returns a thread dump too. Only health is exposed over HTTP by default, so it has to be enabled with management.endpoints.web.exposure.include and secured before the incident, not during it.
  • From inside the JVM, ThreadMXBean.findDeadlockedThreads() finds cycles over both monitors and java.util.concurrent locks for platform threads; it does not find cycles that include virtual threads. A scheduled check that logs the result turns a silent hang into an alert.
  • The immediate mitigation is a restart, because nothing else frees the threads. Take the dump first: after the restart the evidence is gone and the bug may not show again for weeks.
Investigate · PracticeSome transfers hang forever, and more of them every hour

References

References: jcmd, JDK 21 tool specification (Thread.print) ↗ · ThreadMXBean.findDeadlockedThreads (Java SE 21 API) ↗ · ReentrantLock (Java SE 21 API): tryLock with a timeout ↗ · Thread.State (Java SE 21 API): BLOCKED, WAITING, TIMED_WAITING ↗ · Java Language Specification, Java SE 21, chapter 17: Threads and Locks (17.1 synchronization) ↗ · Spring Boot reference: Actuator endpoints (threaddump) ↗ · Report a problem