CompletableFuture starvation: a pool waiting for itself
Four order tasks on a pool of two each call join() on a price task queued on the same pool. Nothing ever finishes. Step through it, read the thread dump, then fix it with thenCompose.
The situation
A Spring Boot 3 order service loads an order and its price asynchronously on one shared executor of 8 threads. It has run for months at 2 to 3 concurrent requests. During a sale, 8 requests arrive within a few milliseconds. From that moment every call to GET /orders/{id} hangs until the gateway's 30 second timeout, including requests that arrive after the burst is over. CPU is at 0%, the heap is flat, the database is idle and there is no exception in the log. A restart fixes it until the next burst.
Read first: CompletableFuture and asynchronous execution, Executors, thread pools and rejection
Break it in the simulation
Two threads, four orders, each waiting for a price
Simulation · not a real JVMPress Step four times with the lab settings and watch both threads end in WAITING. Then change Pool size and Orders to find when it hangs, and switch to thenCompose.
Guided run · 0 of 6 stages done
- Set Orders to 1. Leave Pool size at 2 and How an order gets its price on Blocking join().
- Press Step three times and read What just happened after each press.
Look for
- order-pool-1 runs order 1, submits price 1 and goes to WAITING. order-pool-2 is idle, so it runs price 1.
- Completed is 1 of 1 at Time 50 ms.
- The Result is a warning, not a pass: it only worked because one thread was spare. This is the state the service ran in for months.
- Set Orders to 4 (the lab setting).
- Press Step four times, or Run.
Look for
- Step 1: orders 1 and 2 take both threads, orders 3 and 4 wait in Order pool queue.
- Steps 2 and 3: each running order submits its price task, which goes to the back of the queue, and then calls
join(). - In join() is 2 of 2. Order pool queue holds order 3, order 4, price 1, price 2, and no thread is left to take any of them.
- Completed stays at 0 of 4 and Time stays at 0 ms: the same result as the real run, which gave up after 3 seconds with 0 of 4.
- Stay on the last step and read Thread dump of the pool threads.
- Compare it with the real dump from the lab further down the page.
Look for
- Both pool threads are
WAITING (parking)withCompletableFuture.joinin the stack, called from the order lambda. - They park on a
CompletableFuture$Signaller, not on a monitor or a lock. That is why a thread dump does not print "Found one Java-level deadlock" here: no lock cycle exists. - Nothing in the dump shows the queue. You conclude it: every thread of the pool waits for a future, and only this pool could complete it.
- Set Pool size to 4 and press Run.
- Set Pool size to 5 and press Run.
- Keep Pool size at 5 and set Orders to 5.
Look for
- Pool of 4, 4 orders: still 0 of 4. All four threads wait and the four price tasks sit in the queue.
- Pool of 5, 4 orders: 4 of 4 after 150 ms, through the one spare thread.
- Pool of 5, 5 orders: 0 of 5 again. It hangs whenever concurrent orders reach the pool size.
- Set Pool size back to 2 and Orders to 4.
- Switch How an order gets its price to thenCompose and press Run.
Look for
- The first stage of each order gives its thread back at once. In join() stays at 0 of 2 for the whole run.
- Completed reaches 4 of 4 at Time 100 ms: four price lookups of 50 ms on two threads. The real run took 111 ms.
- The thread dump at the end shows both threads idle, parked in the queue's
take.
- With thenCompose, set Pool size to 1 and Orders to 12 and press Run.
- Set Pool size to 2 and Orders to 4, switch to Separate pool and press Run.
Look for
- One thread, 12 orders: 12 of 12 after 600 ms. Slow, but it cannot hang, because no task waits for another task.
- Separate pool: 4 of 4 after 100 ms as well, but In join() shows order threads WAITING during the run. The deadlock is gone; the blocked threads are not.
Wrong, then right
Incorrect implementation
import java.util.concurrent.CompletableFuture;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import org.springframework.stereotype.Service;@Servicepublic class OrderViewService { // One pool for "everything async" in this service. private final ExecutorService orderPool = Executors.newFixedThreadPool(8); private final OrderRepository orders; private final PricingClient pricing; public OrderViewService(OrderRepository orders, PricingClient pricing) { this.orders = orders; this.pricing = pricing; } public CompletableFuture<OrderView> load(long orderId) { return CompletableFuture.supplyAsync(() -> { Order order = orders.findById(orderId); // A second task on the SAME pool... CompletableFuture<Price> price = CompletableFuture.supplyAsync(() -> pricing.priceFor(order), orderPool); // ...and this pool thread now waits for it. return new OrderView(order, price.join()); }, orderPool); }}Why it is written this way
- The outer task occupies a pool thread from start to finish, including the time it spends inside
join(). - The inner task is submitted to the same pool. If no thread is free it goes into the queue, behind any orders that are already waiting.
- With 8 threads this works for up to 7 concurrent calls and hangs for good at 8. Load tests with low concurrency pass.
- Not compiled here as written (it needs Spring and the service's own types). The same class without the annotation was compiled and run on Java 21 with a pool of 2: 0 of 4 calls finished.
Why it fails
- A thread that calls
join()on an incomplete future parks and stays assigned to its task. The pool counts it as busy. AThreadPoolExecutordoes not add a thread to make up for one that is blocked. - When as many order tasks run as the pool has threads, each one has put its price task into the queue and parked. The price tasks can only be run by threads of this pool, and all of them are waiting for those price tasks. Every thread waits for work that only the waiting threads could do.
- It never recovers. New requests add more order tasks to the queue behind the stuck price tasks, and nothing takes from that queue. The executor's queue grows until the process is restarted.
- No lock is involved, so the JVM's deadlock detection, which looks for cycles of threads waiting for object monitors or ownable synchronizers (such as
ReentrantLock), reports nothing. The service looks idle: 0% CPU, no exception, no log line. - It depends on concurrency, not on load over time. Seven concurrent requests work forever; eight at the same instant stop the service.
Corrected implementation
import java.util.concurrent.CompletableFuture;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import java.util.concurrent.TimeUnit;import org.springframework.stereotype.Service;@Servicepublic class OrderViewService { private final ExecutorService orderPool = Executors.newFixedThreadPool(8); private final OrderRepository orders; private final PricingClient pricing; public OrderViewService(OrderRepository orders, PricingClient pricing) { this.orders = orders; this.pricing = pricing; } public CompletableFuture<OrderView> load(long orderId) { return CompletableFuture .supplyAsync(() -> orders.findById(orderId), orderPool) // Runs when the order is there. Returns a future instead of waiting for one. .thenCompose(order -> CompletableFuture .supplyAsync(() -> pricing.priceFor(order), orderPool) .thenApply(price -> new OrderView(order, price))) // A stuck dependency fails the request instead of holding it forever. .orTimeout(2, TimeUnit.SECONDS); }}Why it is written this way
thenComposetakes a function that returns another stage and completes with that stage's value. The first task ends as soon as the order is loaded and its thread goes back to the pool.- No code in this method calls
join()orget(). That is the rule: a task that runs on a pool must not block on another task of the same pool. thenApply(non-async) runs on the thread that completes the price future, or on the calling thread if the future is already complete. It is a few nanoseconds of work, so that is fine.orTimeoutcompletes the future exceptionally with aTimeoutExceptionafter 2 seconds. It does not interrupt the task that is still running.- Not compiled here as written (it needs Spring and the service's own types). The same class without the annotation was compiled and run on Java 21 with a pool of 2: 4 of 4 calls finished.
Trade-offs
- The code no longer reads top to bottom. Stack traces of a failure show the pool's frames and the lambda, not the method that started the chain, so errors are harder to trace.
- Exceptions arrive wrapped in
CompletionException, and every chain needs its own error handling (exceptionally,handle); atry/catcharound the method catches nothing. - Composition only helps if the whole chain is non-blocking. One
join()inside any stage on the same pool brings the problem back, and it is easy to add one by accident. - A separate pool for the inner task is the smaller change and also removes the deadlock, but each order still parks a thread while it waits, and you now size and monitor two pools.
- On Java 21 a virtual-thread-per-task executor is another way out for blocking code like this: a virtual thread that waits in
join()does not hold a platform thread. It still does not limit how many calls reach the pricing service.
Run it on a real JVM
CfStarvation.java creates a fixed pool of 2 threads and starts 4 order tasks; each needs a price task that sleeps 50 ms on the same pool. With blocking the order task calls join() on the price future; with composed it uses thenCompose. The program waits 3 seconds for all four, prints how many finished, and in the blocking case stays up for 60 seconds so you can take a thread dump.
With Docker (CPU and memory limits)
cd labs/jvm/completablefuture-starvationdocker build -t jvm-lab-completablefuture-starvation .# broken: prints TIMEOUT after 3 s, then stays up for 60 sdocker run --rm --name cf-lab --cpus=2 --memory=512m jvm-lab-completablefuture-starvation CfStarvation blocking# second terminal, during those 60 s: the JVM is process 1 in the containerdocker exec cf-lab jcmd 1 Thread.print# fixed: thenCompose on the same pooldocker run --rm --cpus=2 --memory=512m jvm-lab-completablefuture-starvation CfStarvation composedWithout Docker, on a local JDK 21
java -XX:ActiveProcessorCount=2 CfStarvation.java blocking# second terminal; the program prints its pid on the first linejcmd <pid> Thread.printjava -XX:ActiveProcessorCount=2 CfStarvation.java composedThe program: CfStarvation.java
import java.util.ArrayList;import java.util.List;import java.util.concurrent.*;/** * Lab 8: CompletableFuture execution and thread pool starvation. * * Each "order" task starts a "price" task on the same small pool and waits for it. * * java CfStarvation.java blocking -> outer tasks hold every thread while they join() inner tasks that can never start * java CfStarvation.java composed -> thenCompose: no thread waits, the same pool finishes at once * * While "blocking" hangs: jcmd <pid> Thread.print */public class CfStarvation { public static void main(String[] args) throws Exception { String mode = args.length > 0 ? args[0] : "blocking"; ExecutorService pool = Executors.newFixedThreadPool(2, r -> { Thread t = new Thread(r); t.setName("order-pool-" + t.threadId()); t.setDaemon(true); return t; }); System.out.println("pid " + ProcessHandle.current().pid() + ", mode " + mode + ", pool of 2 threads, 4 orders"); long t0 = System.nanoTime(); List<CompletableFuture<String>> orders = new ArrayList<>(); for (int i = 1; i <= 4; i++) { int id = i; if (mode.equals("blocking")) { orders.add(CompletableFuture.supplyAsync(() -> { CompletableFuture<Integer> price = CompletableFuture.supplyAsync(() -> price(id), pool); return "order " + id + " costs " + price.join(); // blocks a pool thread until another pool thread is free }, pool)); } else { orders.add(CompletableFuture.supplyAsync(() -> id, pool) .thenCompose(o -> CompletableFuture.supplyAsync(() -> price(o), pool)) .thenApply(p -> "order " + id + " costs " + p)); } } try { CompletableFuture.allOf(orders.toArray(CompletableFuture[]::new)).get(3, TimeUnit.SECONDS); for (CompletableFuture<String> o : orders) System.out.println(o.join()); System.out.printf("finished in %d ms%n", (System.nanoTime() - t0) / 1_000_000); } catch (TimeoutException e) { long done = orders.stream().filter(CompletableFuture::isDone).count(); System.out.printf("TIMEOUT after 3 s: %d of 4 orders finished. Both pool threads are waiting in join() for price tasks that sit in the pool's own queue.%n", done); System.out.println("The process will now stay up for 60 s so you can take a thread dump: jcmd " + ProcessHandle.current().pid() + " Thread.print"); Thread.sleep(60_000); } } static int price(int orderId) { try { Thread.sleep(50); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } return 100 + orderId; }}The Dockerfile
# Java 21 lab image. Build: docker build -t jvm-lab-completablefuture-starvation .FROM eclipse-temurin:21-jdkWORKDIR /labCOPY CfStarvation.java .RUN javac -d out CfStarvation.java# The JVM reads the container's CPU and memory limits (container support is on by default),# so --cpus and --memory on "docker run" change what Runtime.availableProcessors() and the default heap size report.ENTRYPOINT ["java", "-cp", "out"]CMD ["CfStarvation"]$ java -XX:ActiveProcessorCount=2 CfStarvation.java blocking
Program outputBlocking join(): 0 of 4 orders finish
Real JVM outputpid 4054, mode blocking, pool of 2 threads, 4 orders TIMEOUT after 3 s: 0 of 4 orders finished. Both pool threads are waiting in join() for price tasks that sit in the pool's own queue. The process will now stay up for 60 s so you can take a thread dump: jcmd 4054 Thread.print
$ jcmd 4054 Thread.print
Thread dumpThread dump: both pool threads WAITING in join()
Real JVM output"order-pool-19" #19 [4074] daemon prio=5 os_prio=0 cpu=0.11ms elapsed=5.74s tid=0x00007384bc58a9a0 nid=4074 waiting on condition [0x00007384957fd000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@21.0.13/Native Method) - parking to wait for <0x00000000f7183c30> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.park(java.base@21.0.13/LockSupport.java:221) at java.util.concurrent.CompletableFuture$Signaller.block(java.base@21.0.13/CompletableFuture.java:1864) at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.13/ForkJoinPool.java:3780) at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.13/ForkJoinPool.java:3725) at java.util.concurrent.CompletableFuture.waitingGet(java.base@21.0.13/CompletableFuture.java:1898) at java.util.concurrent.CompletableFuture.join(java.base@21.0.13/CompletableFuture.java:2117) at CfStarvation.lambda$main$2(CfStarvation.java:33) at CfStarvation$Lambda/0x0000738448164428.get(Unknown Source) at java.util.concurrent.CompletableFuture$AsyncSupply.run(java.base@21.0.13/CompletableFuture.java:1768) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.13/ThreadPoolExecutor.java:1144) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.13/ThreadPoolExecutor.java:642) at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596) at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583) "order-pool-20" #20 [4075] daemon prio=5 os_prio=0 cpu=0.25ms elapsed=5.74s tid=0x00007384bc58b950 nid=4075 waiting on condition [0x00007384956fd000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@21.0.13/Native Method) - parking to wait for <0x00000000f7138e88> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.park(java.base@21.0.13/LockSupport.java:221) at java.util.concurrent.CompletableFuture$Signaller.block(java.base@21.0.13/CompletableFuture.java:1864) at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.13/ForkJoinPool.java:3780) at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.13/ForkJoinPool.java:3725) at java.util.concurrent.CompletableFuture.waitingGet(java.base@21.0.13/CompletableFuture.java:1898) at java.util.concurrent.CompletableFuture.join(java.base@21.0.13/CompletableFuture.java:2117) at CfStarvation.lambda$main$2(CfStarvation.java:33) at CfStarvation$Lambda/0x0000738448164428.get(Unknown Source) at java.util.concurrent.CompletableFuture$AsyncSupply.run(java.base@21.0.13/CompletableFuture.java:1768) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.13/ThreadPoolExecutor.java:1144) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.13/ThreadPoolExecutor.java:642) at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596) at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)
order-pool-19 and order-pool-20, are the whole pool. Both are WAITING (parking) and both stacks read, from the bottom up: ThreadPoolExecutor.runWorker → CompletableFuture$AsyncSupply.run → the order lambda (CfStarvation.lambda$main$2) → CompletableFuture.join. Each parks on a different CompletableFuture$Signaller: its own price future. cpu=0.11ms and cpu=0.25ms after elapsed=5.74s say they have done nothing. The full file has no "Found one Java-level deadlock" section.$ java -XX:ActiveProcessorCount=2 CfStarvation.java composed
Program outputthenCompose: 4 of 4 in 111 ms
Real JVM outputpid 4120, mode composed, pool of 2 threads, 4 orders order 1 costs 101 order 2 costs 102 order 3 costs 103 order 4 costs 104 finished in 111 ms
Real run and simulation: what each one shows
- The real run shows what you would actually have in an incident: a process that is up, silent and idle, and a thread dump in which the proof is two identical stacks ending in
CompletableFuture.join. The dump does not show the pool's queue; the simulation draws it. - The real dump contains every JVM thread (compiler, GC, the main thread sleeping). Finding the two that matter by their name prefix is part of the skill. The simulated dump shows only the pool threads and shortens the frames.
- The simulation lets you change the pool size and the number of orders and see the rule (it hangs when concurrent orders reach the pool size). The lab is fixed at 2 and 4 unless you edit the source.
- The simulation's times are exact multiples of 50 ms. The real composed run took 111 ms, not 100, and that number changes from run to run.
- The captured files come from OpenJDK 21.0.13 run directly on Linux x86-64 with
-XX:ActiveProcessorCount=2. The Docker commands above are the same program in a container; they were not executed when this page was written.
How you meet this in production
- The first symptom is requests that never return: every call to the affected endpoints times out at the gateway while CPU is near zero and the service's health check may still pass. Unlike overload, it does not get better when traffic drops.
- Take a thread dump:
jcmd <pid> Thread.print. Find the pool's threads by name. If all of them areWAITING (parking)withCompletableFuture.joinorCompletableFuture.getin the stack, called from a task of that same pool, this is it. - Do not expect the "Found one Java-level deadlock" section. It appears for cycles of locks, and here the threads wait for futures.
- The executor's numbers show it too: active threads equal to the pool size, completed-task count not moving, queue size rising. An overloaded pool also has a growing queue, but its completed count keeps moving.
- It usually starts with a burst (a sale, a cache expiry, a retry storm after a short outage) on code that has worked for a long time. Name your pool threads: with the default
pool-1-thread-3names the dump is much harder to read.
References
References: CompletableFuture (Java SE 21 API): join, thenCompose, supplyAsync, orTimeout, which thread runs dependent actions ↗ · ThreadPoolExecutor (Java SE 21 API): queuing, and direct handoffs for requests with internal dependencies ↗ · Executors.newFixedThreadPool (Java SE 21 API) ↗ · ThreadMXBean.findDeadlockedThreads (Java SE 21 API): what the JVM's deadlock detection covers ↗ · jcmd (Java SE 21 tool specification): Thread.print ↗ · Report a problem