Thread pool exhaustion: four threads, five times the load
A pool of 4 threads with a queue of 8 gets requests five times faster than it can finish them. Watch the queue fill, compare the rejection policies, and see why an unbounded queue hides the problem.
The situation
A Spring Boot 3 checkout service calls a pricing service on a dedicated executor of 4 threads. The call normally takes 30 ms. After a pricing deploy it takes 200 ms, while checkout traffic stays at 100 requests per second. The pool can now finish 4 ÷ 0.2 s = 20 calls per second, one fifth of what arrives. Users see checkout p99 climb from 60 ms to several seconds and then time out. CPU is low, the heap is flat, and there are no errors in the log, because the executor was created with the default unbounded queue and simply keeps every request waiting.
Read first: Executors, thread pools and rejection
Break it in the simulation
A pool of 4, a queue of 8, 100 requests per second
Simulation · not a real JVMPress Play with the lab settings and watch the queue fill and the Rejected counter rise. Then change Rejection policy and Unbounded queue, and finally bring Utilisation under 100%.
Guided run · 0 of 6 stages done
- Set Arrival rate to 15 requests/s. Leave Threads at 4, Queue capacity at 8, Work time at 200 ms and Rejection policy on abort.
- Press Run to end.
Look for
- Utilisation is 75%: 15 requests/s × 200 ms ÷ 4 threads.
- Worker threads never shows more than 3 of 4 busy and Work queue stays empty.
- Rejected is 0 and Queue wait, max is 0 ms. The verdict says the pool keeps up.
- Set Arrival rate back to 100 requests/s (the lab setting).
- Press Play and watch Pool state. Use Pause and Step to stop on the moment the queue fills.
Look for
- Utilisation is 500%. The pool finishes 20 tasks per second and 100 arrive.
- After about 110 ms all 4 threads are busy and all 8 queue slots are taken. Work queue turns to FULL.
- Rejected ends at 40 of 60 (the real run: 39). Queue wait, max is 400 ms: a task that got the last slot waits two full work times.
- The Progress log has the same shape as the lab output below it on this page.
- Switch Rejection policy to caller-runs.
- Press Play and watch Submitting thread (the caller).
Look for
- Rejected stays 0. Each time the queue is full the submitting thread shows BLOCKED and runs the task itself for 200 ms.
- Caller blocked ends at 1,600 ms (8 tasks run by the caller).
- Time to submit all is 2,190 ms instead of 600 ms (the real run: 2,223 ms). The submitter was slowed to the speed of the pool. That is backpressure.
- Switch Rejection policy back to abort and turn Unbounded queue on.
- Press Run to end. Then set Requests in the burst to 200 and press Run to end again.
Look for
- Rejected is 0 and the caller is never blocked. On a dashboard this looks healthy.
- The queue peaks at 48 waiting tasks (the real run: 47) and Queue wait, max is 2,240 ms (the real run: 2,225 ms).
- With 200 requests the peak is 160 and the max wait is 7,840 ms. Queue depth over time is a straight line upwards for as long as requests arrive: the queue grows by 80 tasks per second.
- A large bounded queue behaves the same for a while: turn Unbounded queue off and set Queue capacity to 100.
- Turn Unbounded queue off, set Queue capacity to 8, Requests in the burst to 60 and Rejection policy to abort.
- First try only more threads: set Threads to 16 and press Run to end.
- Then also set Work time to 120 ms (a timeout or a faster downstream call) and press Run to end.
Look for
- 16 threads at 200 ms: Utilisation is still 125%, and 4 of 60 are rejected. More threads moved the limit but did not remove it.
- 16 threads at 120 ms: Utilisation is 75%, Rejected is 0 and Queue wait, max is 0 ms.
- The other route to the same result: Threads 4 and Work time 30 ms, which is the service before the pricing deploy.
- Keep the fixed settings and set Requests in the burst to 200.
- Press Run to end.
Look for
- Rejected is still 0 and the queue stays empty: under 100% utilisation the result does not depend on how long the load lasts.
- Now raise Work time to 200 ms again: rejections come back within the first second. The bounded queue and the abort policy are what make that visible at once.
Wrong, then right
Incorrect implementation
import java.util.concurrent.CompletableFuture;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;import org.springframework.web.bind.annotation.GetMapping;import org.springframework.web.bind.annotation.PathVariable;import org.springframework.web.bind.annotation.RestController;@Configurationclass PricingExecutorConfig { @Bean ThreadPoolTaskExecutor pricingExecutor() { ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); executor.setCorePoolSize(4); executor.setThreadNamePrefix("pricing-"); // No queue capacity set: the default is Integer.MAX_VALUE. // No max pool size needed either: with an unbounded queue it is never used. return executor; }}@RestControllerclass CheckoutController { private final ThreadPoolTaskExecutor pricingExecutor; private final PricingClient pricing; CheckoutController(ThreadPoolTaskExecutor pricingExecutor, PricingClient pricing) { this.pricingExecutor = pricingExecutor; this.pricing = pricing; } @GetMapping("/checkout/{cartId}/quote") CompletableFuture<Quote> quote(@PathVariable String cartId) { // Always accepted. When pricing is slow the task just waits in the queue. return CompletableFuture.supplyAsync(() -> pricing.price(cartId), pricingExecutor); }}Why it is written this way
ThreadPoolTaskExecutordefaults to a queue capacity ofInteger.MAX_VALUE.Executors.newFixedThreadPool(4)has the same shape: a fixed number of threads on an unbounded queue.- With an unbounded queue the pool never grows past the core size and never rejects, so the rejection policy is dead code.
- Nothing limits how long a request may wait: there is no timeout on the future and none on the pricing call.
- Not compiled here (it needs Spring on the classpath). The same executor and future calls were compiled and run as plain Java 21.
Why it fails
- Capacity is threads ÷ work time: 4 ÷ 0.2 s = 20 tasks per second. Arrivals are 100 per second. Utilisation is 100 × 0.2 ÷ 4 = 500%, so 80 tasks per second go into the queue and stay there.
- The queue has no limit, so
executealways succeeds. No exception is thrown and no error is logged. The only signal is the queue length and the time each task waits, and neither is on a default dashboard. - Wait time grows with the backlog: a task behind 48 others waits 48 ÷ 20 per second = 2.4 s before a thread even starts it. The HTTP client has usually given up by then, so the pool spends its threads on answers nobody reads.
- Each queued task holds its lambda, the future and whatever the request captured. If the overload lasts, the queue becomes the largest thing on the heap and the incident ends as an OutOfMemoryError that looks unrelated to pricing.
- A retrying client makes it worse: a timed-out request is sent again and joins the same queue behind its own abandoned copy.
Corrected implementation
import java.util.concurrent.CompletableFuture;import java.util.concurrent.RejectedExecutionException;import java.util.concurrent.ThreadPoolExecutor;import java.util.concurrent.TimeUnit;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.http.HttpStatus;import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;import org.springframework.web.bind.annotation.GetMapping;import org.springframework.web.bind.annotation.PathVariable;import org.springframework.web.bind.annotation.RestController;import org.springframework.web.server.ResponseStatusException;@Configurationclass PricingExecutorConfig { @Bean ThreadPoolTaskExecutor pricingExecutor() { ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); // 100 requests/s x 0.12 s (the client timeout below) = 12 busy threads at worst. 16 leaves 25% spare. executor.setCorePoolSize(16); executor.setMaxPoolSize(16); // A short queue absorbs a burst of a few milliseconds. 8 tasks wait at most 8 / 133 per second = 60 ms. executor.setQueueCapacity(8); executor.setRejectedExecutionHandler(new ThreadPoolExecutor.AbortPolicy()); executor.setThreadNamePrefix("pricing-"); return executor; }}@RestControllerclass CheckoutController { private final ThreadPoolTaskExecutor pricingExecutor; private final PricingClient pricing; // its HTTP client has a 120 ms response timeout CheckoutController(ThreadPoolTaskExecutor pricingExecutor, PricingClient pricing) { this.pricingExecutor = pricingExecutor; this.pricing = pricing; } @GetMapping("/checkout/{cartId}/quote") CompletableFuture<Quote> quote(@PathVariable String cartId) { try { return CompletableFuture .supplyAsync(() -> pricing.price(cartId), pricingExecutor) .orTimeout(250, TimeUnit.MILLISECONDS); } catch (RejectedExecutionException e) { // Pool and queue are full: answer now instead of letting the request wait. throw new ResponseStatusException(HttpStatus.SERVICE_UNAVAILABLE, "pricing is saturated", e); } }}Why it is written this way
- Size from the load: busy threads = arrival rate × time per task. Bound the time per task with a client timeout, otherwise the sizing means nothing when the downstream slows down.
- The queue is short on purpose. A queue is for bursts, not for a rate the pool cannot serve. Its capacity divided by the pool's rate is the longest wait you accept.
AbortPolicyis already the default; it is written out so that nobody has to look it up. The rejection is thrown bysupplyAsyncon the calling thread, which is why a plaintryaround it works. Spring wraps it inTaskRejectedException, a subclass ofRejectedExecutionException.orTimeoutcompletes the future with aTimeoutExceptionafter 250 ms. It does not stop the task: the pool thread is freed only when the pricing call returns or its own timeout fires.- Not compiled here (it needs Spring on the classpath). The same executor settings, the synchronous rejection and
orTimeoutwere compiled and run as plain Java 21.
Trade-offs
- Some requests now fail fast with 503 during overload. That is the point, but callers must handle it, and a retry without backoff turns the 503s into more load.
- The sizing is tied to numbers that change: traffic, the downstream latency and the timeout. It needs a metric on queue length and rejections, and a review when traffic grows.
- More threads means more concurrent calls into the pricing service. If pricing is slow because it is overloaded, 16 callers instead of 4 can make it slower. A pool is also a limit that protects the downstream.
CallerRunsPolicyis the alternative when nothing may be dropped, for example a batch job. On a web request thread it moves the slow call onto Tomcat's threads and can exhaust that pool instead.- A timeout on the future frees the caller but not the worker thread. Only a timeout on the call itself frees the thread.
Run it on a real JVM
PoolExhaustion.java builds a ThreadPoolExecutor with 4 threads and submits 60 tasks of 200 ms, one every 10 ms. The argument picks the variant: abort (queue of 8, AbortPolicy), caller-runs (queue of 8, CallerRunsPolicy) or unbounded (LinkedBlockingQueue). It prints the pool's own counters after every 10th submit and a summary at the end, so the three runs can be compared line by line.
With Docker (CPU and memory limits)
cd labs/jvm/thread-pool-exhaustiondocker build -t jvm-lab-thread-pool-exhaustion .# broken, visible: bounded queue + AbortPolicy rejects what does not fitdocker run --rm --cpus=2 --memory=512m jvm-lab-thread-pool-exhaustion PoolExhaustion abort# backpressure: the submitting thread runs rejected tasks itselfdocker run --rm --cpus=2 --memory=512m jvm-lab-thread-pool-exhaustion PoolExhaustion caller-runs# broken, hidden: unbounded queue, nothing rejected, the wait growsdocker run --rm --cpus=2 --memory=512m jvm-lab-thread-pool-exhaustion PoolExhaustion unbounded# each run ends within about 3 seconds, so there is no time for a thread dump: the program prints the pool counters itselfWithout Docker, on a local JDK 21
java -XX:ActiveProcessorCount=2 PoolExhaustion.java abortjava -XX:ActiveProcessorCount=2 PoolExhaustion.java caller-runsjava -XX:ActiveProcessorCount=2 PoolExhaustion.java unboundedThe program: PoolExhaustion.java
import java.util.ArrayList;import java.util.List;import java.util.concurrent.*;import java.util.concurrent.atomic.AtomicInteger;/** * Lab 1: thread pool exhaustion and queue saturation. * * A pool of 4 threads with a queue of 8 serves requests that each wait 200 ms on a slow dependency. * Requests arrive every 10 ms, five times faster than the pool can finish them. * * java PoolExhaustion.java abort -> AbortPolicy: fails fast with RejectedExecutionException * java PoolExhaustion.java caller-runs -> CallerRunsPolicy: the submitting thread does the work (backpressure) * java PoolExhaustion.java unbounded -> unbounded queue: nothing is rejected, the wait grows without limit */public class PoolExhaustion { static final int REQUESTS = 60; static final int ARRIVAL_MS = 10; static final int WORK_MS = 200; public static void main(String[] args) throws Exception { String mode = args.length > 0 ? args[0] : "abort"; BlockingQueue<Runnable> queue = mode.equals("unbounded") ? new LinkedBlockingQueue<>() : new ArrayBlockingQueue<>(8); RejectedExecutionHandler handler = mode.equals("caller-runs") ? new ThreadPoolExecutor.CallerRunsPolicy() : new ThreadPoolExecutor.AbortPolicy(); ThreadPoolExecutor pool = new ThreadPoolExecutor(4, 4, 0, TimeUnit.SECONDS, queue, handler); AtomicInteger rejected = new AtomicInteger(); List<Long> queueWaitMs = new CopyOnWriteArrayList<>(); List<Future<?>> futures = new ArrayList<>(); long start = System.nanoTime(); for (int i = 1; i <= REQUESTS; i++) { long submitted = System.nanoTime(); try { futures.add(pool.submit(() -> { queueWaitMs.add((System.nanoTime() - submitted) / 1_000_000); sleep(WORK_MS); // the slow downstream call })); } catch (RejectedExecutionException e) { rejected.incrementAndGet(); } if (i % 10 == 0) { System.out.printf("t=%4d ms submitted=%2d active=%d queued=%2d completed=%2d rejected=%2d%n", (System.nanoTime() - start) / 1_000_000, i, pool.getActiveCount(), pool.getQueue().size(), pool.getCompletedTaskCount(), rejected.get()); } sleep(ARRIVAL_MS); } long submitDone = (System.nanoTime() - start) / 1_000_000; for (Future<?> f : futures) f.get(); pool.shutdown(); long total = (System.nanoTime() - start) / 1_000_000; long max = queueWaitMs.stream().mapToLong(Long::longValue).max().orElse(0); double avg = queueWaitMs.stream().mapToLong(Long::longValue).average().orElse(0); System.out.printf("%nmode=%s%n", mode); System.out.printf("served=%d rejected=%d%n", futures.size(), rejected.get()); System.out.printf("time to submit all %d requests: %d ms (%d ms if nothing blocked the caller)%n", REQUESTS, submitDone, REQUESTS * ARRIVAL_MS); System.out.printf("wait in queue before a thread picked the task up: avg %.0f ms, max %d ms%n", avg, max); System.out.printf("all work finished after %d ms%n", total); } static void sleep(long ms) { try { Thread.sleep(ms); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } }}The Dockerfile
# Java 21 lab image. Build: docker build -t jvm-lab-thread-pool-exhaustion .FROM eclipse-temurin:21-jdkWORKDIR /labCOPY PoolExhaustion.java .RUN javac -d out PoolExhaustion.java# The JVM reads the container's CPU and memory limits (container support is on by default),# so --cpus and --memory on "docker run" change what Runtime.availableProcessors() and the default heap size report.ENTRYPOINT ["java", "-cp", "out"]CMD ["PoolExhaustion"]$ java -XX:ActiveProcessorCount=2 PoolExhaustion.java abort
Program outputAbortPolicy: 39 of 60 rejected
Real JVM outputt= 93 ms submitted=10 active=4 queued= 6 completed= 0 rejected= 0 t= 204 ms submitted=20 active=4 queued= 8 completed= 1 rejected= 7 t= 306 ms submitted=30 active=4 queued= 8 completed= 4 rejected=14 t= 408 ms submitted=40 active=4 queued= 8 completed= 5 rejected=23 t= 509 ms submitted=50 active=4 queued= 8 completed= 8 rejected=30 t= 612 ms submitted=60 active=4 queued= 7 completed=10 rejected=39 mode=abort served=21 rejected=39 time to submit all 60 requests: 622 ms (600 ms if nothing blocked the caller) wait in queue before a thread picked the task up: avg 259 ms, max 397 ms all work finished after 1203 ms
active=4 and queued=8: the pool and the queue are both full, and rejected rises by 7 to 9 for every 10 submits. Only 21 tasks were served. Submitting took 622 ms, so the caller was never slowed down. The longest queue wait is 397 ms, about two work times, because the queue is short.$ java -XX:ActiveProcessorCount=2 PoolExhaustion.java caller-runs
Program outputCallerRunsPolicy: nothing rejected, the submitter slows down
Real JVM outputt= 94 ms submitted=10 active=4 queued= 6 completed= 0 rejected= 0 t= 606 ms submitted=20 active=4 queued= 5 completed= 9 rejected= 0 t= 907 ms submitted=30 active=4 queued= 7 completed=16 rejected= 0 t=1409 ms submitted=40 active=4 queued= 6 completed=25 rejected= 0 t=1711 ms submitted=50 active=4 queued= 8 completed=32 rejected= 0 t=2213 ms submitted=60 active=4 queued= 7 completed=41 rejected= 0 mode=caller-runs served=60 rejected=0 time to submit all 60 requests: 2223 ms (600 ms if nothing blocked the caller) wait in queue before a thread picked the task up: avg 219 ms, max 374 ms all work finished after 2635 ms
rejected stays 0 and all 60 are served. Look at the time column instead: the gap between two lines is no longer 100 ms but 300 to 500 ms, because the submitting thread kept running tasks itself. Submitting all 60 took 2,223 ms instead of 600 ms. Queue wait stays the same as with abort, because the queue is still 8 slots.$ java -XX:ActiveProcessorCount=2 PoolExhaustion.java unbounded
Program outputUnbounded queue: nothing rejected, the wait climbs past 2 seconds
Real JVM outputt= 95 ms submitted=10 active=4 queued= 6 completed= 0 rejected= 0 t= 204 ms submitted=20 active=4 queued=15 completed= 1 rejected= 0 t= 306 ms submitted=30 active=4 queued=22 completed= 4 rejected= 0 t= 407 ms submitted=40 active=4 queued=31 completed= 5 rejected= 0 t= 509 ms submitted=50 active=4 queued=38 completed= 8 rejected= 0 t= 610 ms submitted=60 active=4 queued=47 completed= 9 rejected= 0 mode=unbounded served=60 rejected=0 time to submit all 60 requests: 622 ms (600 ms if nothing blocked the caller) wait in queue before a thread picked the task up: avg 1109 ms, max 2225 ms all work finished after 3036 ms
rejected is 0 and submitting took 622 ms, so from outside this looks like the healthy case. queued is the line to read: 6, 15, 22, 31, 38, 47. It grows by about 8 for every 10 submits. The average queue wait is 1,109 ms and the longest is 2,225 ms, for a task that needs 200 ms of work.Real run and simulation: what each one shows
- The real run shows that this is how
java.util.concurrent.ThreadPoolExecutorbehaves on a real JVM, with real timing noise: 39 rejections where the model gives 40, 2,223 ms where the model gives 2,190 ms, a queue of 47 where the model gives 48.Thread.sleep(10)takes a little more than 10 ms, so the real submitter is slightly slower than the model's. - The simulation shows what the lab cannot: every moment between the printed lines, each thread and each queue slot, and what happens when you change the pool size, the queue, the rate or the work time without editing and re-running the program.
- The simulation's numbers are exact for its own rules and repeat on every run. The real numbers change by a few requests and milliseconds from run to run and from machine to machine.
- The captured files come from OpenJDK 21.0.13 run directly on Linux x86-64 with
-XX:ActiveProcessorCount=2. The Docker commands above are the same program in a container; they were not executed when this page was written.
How you meet this in production
- The first symptom is latency, not errors: p99 of the endpoint climbs steadily while CPU stays low and the downstream's own latency looks only moderately worse. With an unbounded queue there is nothing in the log.
- Look at the executor's numbers:
ThreadPoolExecutor.getActiveCount()equal to the pool size andgetQueue().size()above zero and rising. Spring Boot Actuator instrumentsThreadPoolTaskExecutorbeans and tags the metrics with the bean name, so put queue size and active threads on the service dashboard. - A thread dump (
jcmd <pid> Thread.print) confirms it: everypricing-thread is inside the downstream call (a socket read), none is parked waiting for work. Idle pool threads would be parked in the queue'stake. - With a bounded queue and
AbortPolicythe symptom is a burst ofRejectedExecutionException(in Spring,TaskRejectedException) or 503 responses. That is the healthier failure: it points at the pool by name. - The Practice case Checkout latency climbs walks through this diagnosis from the dashboards.
References
References: ThreadPoolExecutor (Java SE 21 API): core and maximum pool sizes, queuing, rejected tasks ↗ · Executors.newFixedThreadPool (Java SE 21 API): a fixed number of threads on a shared unbounded queue ↗ · jcmd (Java SE 21 tool specification): Thread.print ↗ · Spring Framework: ThreadPoolTaskExecutor (default queue capacity, core and max pool size) ↗ · Spring Boot: Actuator metrics, task execution and scheduling metrics ↗ · Report a problem