Skip to content

Path 4 · lesson 1 of 5

Intermediate28 minJava 21 (OpenJDK 21) · Spring Boot 3.5 (Spring Framework 6.2, Micrometer 1.15)

Executors, thread pools and rejection

How ThreadPoolExecutor decides between a thread, the queue and rejection, why an unbounded queue hides overload, and how to find a starved pool in a thread dump.

What you will learn

  • State the exact order in which ThreadPoolExecutor handles a submitted task and predict the pool size and queue length after each submission
  • Explain why Executors.newFixedThreadPool never rejects and what that costs when arrivals outrun the pool
  • Choose between AbortPolicy, CallerRunsPolicy, DiscardPolicy and DiscardOldestPolicy for a given kind of work
  • Size a pool and its queue from the arrival rate, the task time and the wait you are willing to accept
  • Configure ThreadPoolTaskExecutor, @Async and the Tomcat request pool in Spring Boot 3.x and know their defaults
  • Recognise a starved pool from executor metrics and from the pool's threads in a thread dump

Before you start

The concept

An ExecutorService separates submitting work from running it. You hand it a Runnable or Callable; it decides which thread runs the task and when. The implementation behind almost every pool of platform threads in a Java service is ThreadPoolExecutor: a set of worker threads that take tasks from one shared BlockingQueue.

A ThreadPoolExecutor is fully described by five things. Every production question about a pool comes back to one of them.

ParameterWhat it isWhat it decides
corePoolSizeThreads the pool creates before it starts queuingHow much work runs at once in normal operation
maximumPoolSizeUpper limit on threads, used only when the queue is fullExtra capacity for bursts, if the queue is bounded
workQueueWhere tasks wait when corePoolSize threads are busyHow much waiting work the pool accepts, and so how long a task can wait
keepAliveTimeHow long a thread above core may sit idle before it endsHow fast the pool shrinks after a burst
RejectedExecutionHandlerWhat happens to a task the pool cannot takeWho pays for overload: the caller, the task, or nobody visibly

The factory methods in Executors are fixed choices of those five parameters. Executors.newFixedThreadPool(n) is n core threads, n maximum threads and a LinkedBlockingQueue with no capacity argument, which means a capacity of Integer.MAX_VALUE. Its Javadoc says so directly: the threads operate "off a shared unbounded queue". Executors.newCachedThreadPool() is the opposite choice: zero core threads, no limit on threads and a SynchronousQueue that holds nothing, so every task that finds no idle thread starts a new one.

A pool of 4 with an unbounded queue

Simulation · not a real JVM

Requests arrive five times faster than four threads can finish them. Nothing is rejected. Watch the queue length and the wait of the newest task.

Why it exists

A platform thread is an operating system thread with its own stack. Creating one per request works until a slow dependency makes requests pile up: then the service has thousands of threads, each holding a stack, a database connection or a socket, and the dependency that was slow is now flooded. A pool puts a number on how much work runs at once. That number is the pool's real job: it is a concurrency limit, and the queue and the rejection policy say what happens to the work above the limit.

There are only three things a system can do with work it cannot run now: make it wait, refuse it, or slow the sender down. An unbounded queue chooses waiting without limit. A bounded queue with AbortPolicy chooses refusing. CallerRunsPolicy chooses slowing the sender. No setting makes the overload disappear; the settings choose where it shows.

Unbounded queue

  • No task is ever rejected, so no error appears anywhere
  • Wait time grows for as long as arrivals exceed capacity
  • Callers time out on their side while the task is still queued, and the pool then does work nobody is waiting for
  • Each queued task holds its captured objects on the heap until it runs

Bounded queue with a rejection policy

  • The longest possible wait is known: queue capacity divided by the pool's throughput
  • Overload becomes visible at once as rejections or as a slower caller
  • The caller can answer quickly with a 503 or a fallback
  • You have to pick the capacity and handle the rejection in code

In a request and response service the first symptom of a starved pool is not an exception. It is latency that climbs while CPU stays low: threads are all waiting on something slow, new work waits behind them, and the machine looks idle. The checkout latency case in Practice is this failure.

How it works inside

The order, step by step

execute(task) makes at most three attempts, in this order. The Javadoc states the same rules under "Queuing", and submit and invokeAll go through execute.

  1. Fewer than corePoolSize threads exist: start a new thread and give it this task as its first task. This happens even if other threads are idle.
  2. Otherwise offer the task to the queue. If the queue accepts it, the task waits there until a worker takes it.
  3. The queue refused it: start a new thread, unless that would exceed maximumPoolSize.
  4. That failed too: the pool is saturated. Call the RejectedExecutionHandler. The same handler is called for any task submitted after shutdown().

Program output8 one-second tasks into core 2, max 4, queue 2 (java ExecutorsThreadPools.java order)

Real JVM output
corePoolSize=2 maximumPoolSize=4 queue=ArrayBlockingQueue(2) handler=AbortPolicy
task 1 -> new core thread                          poolSize=1 queued=0
task 2 -> new core thread                          poolSize=2 queued=0
task 3 -> queued                                   poolSize=2 queued=1
task 4 -> queued                                   poolSize=2 queued=2
task 5 -> new thread above core (queue was full)   poolSize=3 queued=2
task 6 -> new thread above core (queue was full)   poolSize=4 queued=2
  java.util.concurrent.RejectedExecutionException: Task ExecutorsThreadPools$Lambda/0x00007a6f4c000c30@7adf9f5f rejected from java.util.concurrent.ThreadPoolExecutor@1540e19d[Running, pool size = 4, active threads = 4, queued tasks = 2, completed tasks = 0]
task 7 -> REJECTED                                 poolSize=4 queued=2
task 8 -> REJECTED                                 poolSize=4 queued=2
java.util.concurrent.ThreadPoolExecutor@1540e19d[Running, pool size = 4, active threads = 4, queued tasks = 2, completed tasks = 0]
after shutdown(): isShutdown=true isTerminated=false (queued tasks still run)
execute() after shutdown(): rejected, pool state is Shutting down
awaitTermination(5 s)=true completed=6 isTerminated=true
Tasks 1 and 2 each start a core thread. Tasks 3 and 4 go to the queue although only 2 of the 4 allowed threads exist. Only tasks 5 and 6, which find the queue full, start threads 3 and 4. Tasks 7 and 8 are rejected, and the exception message prints the pool's state: pool size = 4, active threads = 4, queued tasks = 2. Note the consequence: tasks 5 and 6 arrived later than 3 and 4 but start running before them. After shutdown() the 6 accepted tasks still finish; only new ones are refused.

Two details follow from this order. First, threads are created on demand: a new pool has zero threads until tasks arrive, unless you call prestartAllCoreThreads(). Second, a pool with a large bounded queue and a higher maximum grows late: the extra threads appear only after the queue is already full, which is after latency has already suffered.

Three kinds of queue

QueueBehaviourConsequence
SynchronousQueue (direct handoff)Holds nothing. An offer succeeds only if a thread is waiting to take it.Every task either starts at once or creates a thread or is rejected. Needs a high maximum, which allows unbounded thread growth.
LinkedBlockingQueue with no capacityAccepts everything (capacity Integer.MAX_VALUE).Never more than corePoolSize threads. Never a rejection. Unbounded wait and unbounded memory under sustained overload.
ArrayBlockingQueue(n) or LinkedBlockingQueue(n)Accepts n waiting tasks, then refuses.Bounded wait. Rejection policy decides what overload looks like.
The three strategies named in the ThreadPoolExecutor Javadoc: direct handoffs, unbounded queues, bounded queues.

The four rejection policies

PolicyWhat it doesUse it when
AbortPolicy (the default)Throws RejectedExecutionException to the thread that called execute.The caller can fail fast or fall back: an HTTP request that can return 503.
CallerRunsPolicyThe submitting thread runs the task itself, unless the pool is shut down, in which case the task is discarded.The work must not be lost and slowing the submitter is acceptable: a batch loop or a message listener.
DiscardPolicyDrops the task silently.Rarely. Only when nobody ever waits for the result, such as a best-effort refresh. A Future for a discarded task never completes.
DiscardOldestPolicyDrops the task at the head of the queue and retries the new one.Rarely. The Javadoc calls it "rarely acceptable". It fits only when a newer task makes an older one worthless.

Sizing

Size the pool from what the tasks do, not from a habit.

  • CPU-bound work (parsing, compression, calculation): more threads than cores gives no extra throughput, only context switches. Start with the number of available processors, which in a container is the CPU limit, not the host's core count. Check it with Runtime.getRuntime().availableProcessors().
  • Blocking work (HTTP calls, JDBC, file I/O): a thread spends most of its time waiting, so the pool needs about arrival rate × time per task threads to keep up. 100 calls per second that each take 200 ms keep 100 × 0.2 = 20 threads busy on average. Add headroom for variance, then check the limit of what you call: 40 threads are pointless against a database pool of 10 connections.
  • The queue is sized from the wait you accept. A full queue of q tasks in front of n threads that each take t seconds delays the last task by about q × t / n. A queue of 8 before 4 threads at 200 ms is about 400 ms of extra wait at worst.
  • Do not mix CPU-bound and blocking tasks in one pool. A few slow blocking tasks occupy every thread and the short tasks starve behind them. Give each kind of dependency its own pool so that one slow dependency cannot take the threads of another.

The lab's numbers make the arithmetic concrete. Four threads at 200 ms per task finish 20 tasks per second. Arrivals are one every 10 ms, 100 per second. The pool is short by 80 tasks per second, and the three configurations in the next section differ only in what they do with those 80.

Shutdown

  • shutdown() stops accepting new tasks and lets running and queued tasks finish. It returns at once; it does not wait.
  • awaitTermination(timeout, unit) blocks until all tasks are done or the timeout passes, and returns false on timeout.
  • shutdownNow() interrupts the running tasks and returns the queued tasks that never started. A task that ignores interruption keeps running.
  • Since Java 19 ExecutorService is AutoCloseable: close() calls shutdown() and then waits for termination, so try-with-resources works for short-lived executors.
PartnerPool.javaJava
import java.util.List;import java.util.concurrent.ArrayBlockingQueue;import java.util.concurrent.ThreadFactory;import java.util.concurrent.ThreadPoolExecutor;import java.util.concurrent.TimeUnit;import java.util.concurrent.atomic.AtomicInteger;public final class PartnerPool {    public static ThreadPoolExecutor create() {        AtomicInteger n = new AtomicInteger();        ThreadFactory named = r -> new Thread(r, "partner-exec-" + n.incrementAndGet());        return new ThreadPoolExecutor(                16, 16,                          // core = max: a fixed pool                0, TimeUnit.SECONDS,             // no threads above core, so keep-alive is unused                new ArrayBlockingQueue<>(32),    // at most 32 tasks waiting                named,                new ThreadPoolExecutor.AbortPolicy());    }    /** Stop taking work, give running tasks 20 s, then interrupt what is left. */    public static void stop(ThreadPoolExecutor pool) {        pool.shutdown();        try {            if (!pool.awaitTermination(20, TimeUnit.SECONDS)) {                List<Runnable> neverStarted = pool.shutdownNow();                System.err.println("pool did not drain, dropped " + neverStarted.size() + " queued tasks");                pool.awaitTermination(5, TimeUnit.SECONDS);            }        } catch (InterruptedException e) {            pool.shutdownNow();            Thread.currentThread().interrupt();        }    }}

Why it is written this way

  • Naming the threads is not cosmetic. The prefix is how you find this pool in a thread dump and tell it from http-nio-8080-exec- and task- threads.
  • Core equal to max with a bounded queue is the configuration that is easiest to reason about: 16 running, 32 waiting, the 49th concurrent task is rejected.
  • stop is the two-phase shutdown: shutdown(), wait, then shutdownNow() for what is left. Without the wait, a deploy cuts off work that was accepted.
  • Compiled with javac 21.

Architecture

Drawing diagram…

Flowchart. A submitted task first checks whether fewer than corePoolSize threads exist, and if so starts a core thread. Otherwise it is offered to the queue. If the queue is full and fewer than maximumPoolSize threads exist, a new thread starts. Otherwise the rejection handler runs, with one branch for each of the four policies.

flowchart TD
  A["execute(task)"] --> B{"Fewer than corePoolSize threads"}
  B -->|"Yes"| C["Start a core thread with this task"]
  B -->|"No"| D{"Queue accepts the task"}
  D -->|"Yes"| E["Task waits in the queue"]
  D -->|"No, queue is full"| F{"Fewer than maximumPoolSize threads"}
  F -->|"Yes"| G["Start a thread above core"]
  F -->|"No"| H["RejectedExecutionHandler"]
  H --> I["AbortPolicy throws to the caller"]
  H --> J["CallerRunsPolicy runs it in the caller"]
  H --> K["DiscardPolicy drops it"]
  H --> L["DiscardOldestPolicy drops the queue head and retries"]
The path of one submitted task through ThreadPoolExecutor.execute.
  • Read it top to bottom: the queue sits between core and max. That single fact explains why a pool configured as core 10, max 50 with an unbounded queue never has more than 10 threads.
  • The right-hand exit of the second decision is the only way to reach the third. With an unbounded queue the diagram ends at "Task waits in the queue" for every task, for ever.
  • The handler runs in the submitting thread. AbortPolicy throws there and CallerRunsPolicy runs the task there. That is why the choice of policy depends on who the caller is.
  • Workers leave the picture after their first task: each one loops, taking the next task from the queue. An idle worker is a thread parked inside the queue's take().

Step by step

Drawing diagram…

Sequence diagram with a request thread, the pool, its work queue, a worker thread and the rejection handler. Task 1 starts a core thread. Task 5 is offered to the queue and accepted, and the worker takes it when task 1 ends. Task 13 finds the queue full and the pool at its maximum, so the handler throws RejectedExecutionException back to the request thread.

sequenceDiagram
  participant R as Request thread
  participant P as ThreadPoolExecutor
  participant Q as Work queue
  participant W as Worker thread
  participant H as Rejection handler
  R->>P: execute task 1
  P->>W: Start core thread with task 1
  R->>P: execute task 5 while all core threads are busy
  P->>Q: offer task 5
  Q-->>P: accepted
  Note over W: Task 1 ends
  W->>Q: take next task
  Q-->>W: task 5
  R->>P: execute task 13 with queue full and max reached
  P->>H: rejectedExecution
  H-->>R: RejectedExecutionException
  Note over R: The request answers 503 instead of waiting
One task started directly, one queued and later taken by a worker, one rejected back to the caller.
  1. The pool never pushes a queued task to a worker. Workers pull: when a task ends, the worker calls take() on the queue. A task's wait in the queue is therefore the time until some worker finishes what it is doing.
  2. The rejection is synchronous. execute throws in the request thread before it returns, so the caller knows immediately and can answer with an error or a fallback.
  3. Nothing in this exchange measures the wait in the queue. If you want that number, record the submit time in the task and subtract it when the task starts, as the lab does.

In a real system

The lab reproduces a service that calls a slow dependency through a pool of 4 threads. Each call takes 200 ms and a new request arrives every 10 ms, so 60 requests arrive in 600 ms while the pool can finish only 12 in that time. The same program was run with three configurations.

Program outputUnbounded queue (java PoolExhaustion.java unbounded)

Real JVM output
t=  95 ms  submitted=10  active=4  queued= 6  completed= 0  rejected= 0
t= 204 ms  submitted=20  active=4  queued=15  completed= 1  rejected= 0
t= 306 ms  submitted=30  active=4  queued=22  completed= 4  rejected= 0
t= 407 ms  submitted=40  active=4  queued=31  completed= 5  rejected= 0
t= 509 ms  submitted=50  active=4  queued=38  completed= 8  rejected= 0
t= 610 ms  submitted=60  active=4  queued=47  completed= 9  rejected= 0

mode=unbounded
served=60  rejected=0
time to submit all 60 requests: 622 ms (600 ms if nothing blocked the caller)
wait in queue before a thread picked the task up: avg 1109 ms, max 2225 ms
all work finished after 3036 ms
rejected stays 0 and queued climbs by about 8 every 100 ms: 6, 15, 22, 31, 38, 47. That slope is the gap between arrivals and capacity. No error was raised, yet the average wait before a thread even started a task was 1,109 ms and the worst was 2,225 ms, for work that takes 200 ms. In a service this is the graph where p99 latency climbs while the error rate stays at zero.

Program outputQueue of 8 with AbortPolicy (java PoolExhaustion.java abort)

Real JVM output
t=  93 ms  submitted=10  active=4  queued= 6  completed= 0  rejected= 0
t= 204 ms  submitted=20  active=4  queued= 8  completed= 1  rejected= 7
t= 306 ms  submitted=30  active=4  queued= 8  completed= 4  rejected=14
t= 408 ms  submitted=40  active=4  queued= 8  completed= 5  rejected=23
t= 509 ms  submitted=50  active=4  queued= 8  completed= 8  rejected=30
t= 612 ms  submitted=60  active=4  queued= 7  completed=10  rejected=39

mode=abort
served=21  rejected=39
time to submit all 60 requests: 622 ms (600 ms if nothing blocked the caller)
wait in queue before a thread picked the task up: avg 259 ms, max 397 ms
all work finished after 1203 ms
queued stops at 8 and rejected grows instead: 39 of 60 requests were refused. The 21 that were accepted waited at most 397 ms, close to the 8 × 200 / 4 = 400 ms the sizing rule predicts, and the caller was never slowed down (622 ms to submit against 600 ms). Overload is now an explicit, fast failure the caller can handle.

Program outputQueue of 8 with CallerRunsPolicy (java PoolExhaustion.java caller-runs)

Real JVM output
mode=caller-runs
served=60  rejected=0
time to submit all 60 requests: 2223 ms (600 ms if nothing blocked the caller)
wait in queue before a thread picked the task up: avg 219 ms, max 374 ms
all work finished after 2635 ms
Nothing was rejected and the queue wait stayed bounded (max 374 ms), but submitting the 60 requests took 2,223 ms instead of 600 ms. Each time the queue was full, the submitting thread ran a 200 ms task itself and could not submit anything else meanwhile. The submitter was slowed to the rate the pool can sustain. That is the point of the policy, and also its cost.

Finding a starved pool in a thread dump

Metrics tell you a pool is full. A thread dump tells you what its threads are doing. Take one with jcmd <pid> Thread.print and look at the threads with the pool's name prefix. The second program keeps a pool of 4 busy calling a server that accepts the connection and never answers, next to an idle pool of 2.

Program outputThreadPoolExecutor.toString() for both pools

Real JVM output
checkout java.util.concurrent.ThreadPoolExecutor@2f92e0f4[Running, pool size = 4, active threads = 4, queued tasks = 26, completed tasks = 0]
reports  java.util.concurrent.ThreadPoolExecutor@5305068a[Running, pool size = 2, active threads = 0, queued tasks = 0, completed tasks = 0]
toString() is the cheapest diagnostic there is: pool size, active threads, queued tasks and completed tasks. The checkout pool has 4 of 4 threads active, 26 tasks queued and 0 completed after half a second. A completed count that does not move is the difference between a busy pool and a stuck one.

Thread dumpjcmd <pid> Thread.print: an idle worker and a stuck worker

Real JVM output
636259:
2026-11-15 21:41:35
Full thread dump OpenJDK 64-Bit Server VM (21.0.13+9-LTS mixed mode, sharing)
...
"report-exec-1" #19 [636455] prio=5 os_prio=0 cpu=0.38ms elapsed=3.31s tid=0x00007ad10813e1e0 nid=636455 waiting on condition  [0x00007ad09a9fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21.0.13/Native Method)
	- parking to wait for  <0x00000000ffb9efc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@21.0.13/LockSupport.java:371)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.13/AbstractQueuedSynchronizer.java:519)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.13/ForkJoinPool.java:3780)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.13/ForkJoinPool.java:3725)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.13/AbstractQueuedSynchronizer.java:1746)
	at java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.13/LinkedBlockingQueue.java:435)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.13/ThreadPoolExecutor.java:1070)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.13/ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.13/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)
...
"checkout-exec-1" #21 [636465] prio=5 os_prio=0 cpu=0.26ms elapsed=3.30s tid=0x00007ad1081403d0 nid=636465 runnable  [0x00007ad09a7fe000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.SocketDispatcher.read0(java.base@21.0.13/Native Method)
	at sun.nio.ch.SocketDispatcher.read(java.base@21.0.13/SocketDispatcher.java:48)
	at sun.nio.ch.NioSocketImpl.tryRead(java.base@21.0.13/NioSocketImpl.java:256)
	at sun.nio.ch.NioSocketImpl.implRead(java.base@21.0.13/NioSocketImpl.java:307)
	at sun.nio.ch.NioSocketImpl.read(java.base@21.0.13/NioSocketImpl.java:346)
	at sun.nio.ch.NioSocketImpl$1.read(java.base@21.0.13/NioSocketImpl.java:796)
	at java.net.Socket$SocketInputStream.read(java.base@21.0.13/Socket.java:1099)
	at java.net.Socket$SocketInputStream.read(java.base@21.0.13/Socket.java:1093)
	at ExecutorsThreadPools.callInventory(ExecutorsThreadPools.java:98)
	at ExecutorsThreadPools.lambda$dump$3(ExecutorsThreadPools.java:81)
	at ExecutorsThreadPools$Lambda/0x00007ad09c004c00.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.13/ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.13/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith(java.base@21.0.13/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.13/Thread.java:1583)
report-exec-1 is an idle worker: WAITING (parking) with LinkedBlockingQueue.take called from ThreadPoolExecutor.getTask. Threads in getTask are spare capacity. checkout-exec-1 is busy and stuck: below runWorker there is application code, ExecutorsThreadPools.callInventory, and above it SocketInputStream.read. Its state is RUNNABLE although it uses no CPU (cpu=0.26ms after elapsed=3.30s): a thread blocked in a native socket read is reported as RUNNABLE. In the full dump all four checkout-exec- threads show this same stack. Every thread of a pool in the same frame of the same downstream call is the signature of thread starvation, and the frame names the dependency.

In a Spring Boot service the picture has two layers. The Tomcat request threads wait for the pool, and the pool's threads wait for the dependency. A request thread that waits on a future looks like this:

Thread dumpA Tomcat request thread waiting for a pool task

Illustrative, not from a run
"http-nio-8080-exec-17" #58 [70412] daemon prio=5 os_prio=0 cpu=312.44ms elapsed=1841.20s tid=0x00007f3a2c0b1a10 nid=70412 waiting on condition  [0x00007f39d4dfc000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21.0.13/Native Method)
	- parking to wait for  <0x00000000e1a40d18> (a java.util.concurrent.CompletableFuture$Signaller)
	at java.util.concurrent.locks.LockSupport.park(java.base@21.0.13/LockSupport.java:221)
	at java.util.concurrent.CompletableFuture$Signaller.block(java.base@21.0.13/CompletableFuture.java:1864)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.13/ForkJoinPool.java:3780)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.13/ForkJoinPool.java:3725)
	at java.util.concurrent.CompletableFuture.waitingGet(java.base@21.0.13/CompletableFuture.java:1898)
	at java.util.concurrent.CompletableFuture.join(java.base@21.0.13/CompletableFuture.java:2117)
	at com.example.shipping.ShippingQuoteController.quotes(ShippingQuoteController.java:47)
...
Written for this lesson in the format of a JDK 21 dump; the JDK frames match the real dumps in the labs. Count the threads in each group: if nearly all http-nio-8080-exec- threads are parked in join() or get() under one controller method, and all threads of the pool they wait for are in one downstream call, the request pool is not the cause. Raising server.tomcat.threads.max would only let more requests wait.
  • Latency up, CPU low, errors flat: suspect waiting, not computing. Check the executor's active count against its size and whether the queue is growing.
  • Pool threads all in the same downstream frame: the dependency is slow or has no timeout. Fix the timeout first, then the pool.
  • Pool threads idle in getTask while requests are slow: this pool is not the bottleneck. Look at the request pool, the connection pool or a lock.
  • Pool threads parked in join() or get() on tasks of the same pool: the pool is waiting for itself. That case is the subject of the next lesson.

Java and Spring Boot

A Spring Boot 3.5 service with two kinds of background work: carrier calls that may fail fast, and audit records that must not be lost. Each gets its own named, bounded pool. The Spring files were written against Spring Boot 3.5 and Spring Framework 6.2 and were not compiled here.

src/main/resources/application.ymlYAML
server:  tomcat:    threads:      max: 200          # request threads (the default)      min-spare: 10     # the default    accept-count: 100   # connections the OS may queue when every thread is busy (the default)    mbeanregistry:      enabled: true     # needed for the tomcat.threads.* metricsspring:  task:    execution:          # Spring Boot's own executor, bean name applicationTaskExecutor      mode: force       # keep it although this service defines its own Executor beans      thread-name-prefix: app-task-      pool:        core-size: 8    # the default        max-size: 16        queue-capacity: 100        keep-alive: 10s      shutdown:        await-termination: true        await-termination-period: 20smanagement:  endpoints:    web:      exposure:        include: health,metrics,threaddump

Why it is written this way

  • Out of the box the auto-configured executor has 8 core threads and an unbounded queue, so max-size is ignored and nothing is ever rejected. Setting queue-capacity is what turns max-size on.
  • With queue-capacity: 100 the pool grows from 8 to 16 threads only after 100 tasks are already waiting. If you want 16 threads working before anything queues, set core-size: 16.
  • Spring Boot backs off from creating this executor when the application registers an Executor bean of its own. This service defines two below, so mode: force is needed to keep the auto-configured one as well; add spring.task.execution.mode: force if MVC async requests and unnamed @Async methods should still use it.
  • await-termination makes the executor wait for accepted tasks during shutdown, up to the period, instead of abandoning them. Both are off by default.
  • The threaddump Actuator endpoint returns the same information as jcmd Thread.print over HTTP. Do not expose it publicly.

Wrong, then right

A shipping service asks three carriers for a quote on every request. Normal traffic is 40 requests per second and a carrier answers in about 150 ms, so 120 calls per second keep about 18 threads busy. The first version works for months, then one carrier slows down to 2 seconds per call. Neither version was compiled here; both are written against Spring Boot 3.5.

Incorrect implementation

src/main/java/com/example/shipping/ShippingQuoteController.javaJava
package com.example.shipping;import java.util.ArrayList;import java.util.List;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import java.util.concurrent.Future;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.web.bind.annotation.GetMapping;import org.springframework.web.bind.annotation.PathVariable;import org.springframework.web.bind.annotation.RestController;import org.springframework.web.client.RestClient;@Configurationclass CarrierConfig {    @Bean(destroyMethod = "shutdown")    ExecutorService carrierExecutor() {        return Executors.newFixedThreadPool(20);       // unbounded queue, threads named pool-1-thread-N    }    @Bean    RestClient carriers(RestClient.Builder builder) {        return builder.baseUrl("https://carriers.internal.example").build();   // no timeouts    }}@RestControllerclass ShippingQuoteController {    private static final List<String> CARRIERS = List.of("north", "rapid", "blue");    private final ExecutorService carrierExecutor;    private final RestClient carriers;    ShippingQuoteController(ExecutorService carrierExecutor, RestClient carriers) {        this.carrierExecutor = carrierExecutor;        this.carriers = carriers;    }    @GetMapping("/quotes/{orderId}")    List<Quote> quotes(@PathVariable String orderId) throws Exception {        List<Future<Quote>> pending = new ArrayList<>();        for (String carrier : CARRIERS) {            pending.add(carrierExecutor.submit(() -> carriers.get()                    .uri("/{carrier}/quotes/{orderId}", carrier, orderId)                    .retrieve()                    .body(Quote.class)));        }        List<Quote> quotes = new ArrayList<>();        for (Future<Quote> future : pending) {            quotes.add(future.get());                  // waits for as long as it takes        }        return quotes;    }}

Why it is written this way

  • Executors.newFixedThreadPool(20) is a ThreadPoolExecutor with 20 core threads, 20 maximum threads and an unbounded LinkedBlockingQueue. It cannot reject.
  • The RestClient has no connect or read timeout, so one call holds a pool thread for as long as the carrier keeps the connection open.
  • future.get() has no timeout either, so the Tomcat request thread waits for the pool thread, which waits for the carrier.
  • The pool's threads are called pool-1-thread-7. In a dump with several pools nobody can tell which is which.

Why it fails

  • With one carrier at 2 seconds, a third of the 120 calls per second take 2 s instead of 0.15 s. Those 40 slow calls per second alone need 40 × 2 = 80 threads. The pool has 20, so within a second every pool thread is inside a call to the slow carrier.
  • The queue is unbounded, so submit keeps succeeding. Calls to the two healthy carriers now wait in the same queue behind the slow ones. The queue grows by tens of tasks per second and no exception is thrown anywhere.
  • Each request thread sits in future.get() with no timeout. Tomcat has 200 request threads by default and 40 requests arrive per second, so in about 5 seconds every request thread is parked. New connections wait in the accept queue and clients start to time out, including clients of endpoints that never touch a carrier.
  • What you see in production: p99 latency rising from 200 ms to many seconds, CPU falling, zero errors in the service's own logs, heap usage climbing with the queue, and a thread dump with 20 pool-1-thread-N threads in SocketInputStream.read and 200 http-nio-8080-exec-N threads in FutureTask.get.
  • Recovery is slow too. When the carrier comes back, the pool first works through the backlog of calls whose callers gave up long ago.

Corrected implementation

src/main/java/com/example/shipping/ShippingQuoteController.javaJava
package com.example.shipping;import java.net.http.HttpClient;import java.time.Duration;import java.util.List;import java.util.Objects;import java.util.concurrent.CompletableFuture;import java.util.concurrent.TimeUnit;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.core.task.TaskRejectedException;import org.springframework.http.HttpHeaders;import org.springframework.http.HttpStatus;import org.springframework.http.ResponseEntity;import org.springframework.http.client.JdkClientHttpRequestFactory;import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;import org.springframework.web.bind.annotation.GetMapping;import org.springframework.web.bind.annotation.PathVariable;import org.springframework.web.bind.annotation.RestController;import org.springframework.web.client.RestClient;@Configurationclass CarrierConfig {    @Bean    ThreadPoolTaskExecutor carrierExecutor() {        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();        executor.setThreadNamePrefix("carrier-exec-");        executor.setCorePoolSize(24);        executor.setMaxPoolSize(24);        executor.setQueueCapacity(48);                 // AbortPolicy is the default handler        executor.setWaitForTasksToCompleteOnShutdown(true);        executor.setAwaitTerminationSeconds(10);        return executor;    }    @Bean    RestClient carriers(RestClient.Builder builder) {        HttpClient http = HttpClient.newBuilder().connectTimeout(Duration.ofMillis(500)).build();        JdkClientHttpRequestFactory factory = new JdkClientHttpRequestFactory(http);        factory.setReadTimeout(Duration.ofMillis(800));        return builder.baseUrl("https://carriers.internal.example").requestFactory(factory).build();    }}@RestControllerclass ShippingQuoteController {    private static final List<String> CARRIERS = List.of("north", "rapid", "blue");    private final ThreadPoolTaskExecutor carrierExecutor;    private final RestClient carriers;    ShippingQuoteController(ThreadPoolTaskExecutor carrierExecutor, RestClient carriers) {        this.carrierExecutor = carrierExecutor;        this.carriers = carriers;    }    @GetMapping("/quotes/{orderId}")    ResponseEntity<List<Quote>> quotes(@PathVariable String orderId) {        List<CompletableFuture<Quote>> pending;        try {            pending = CARRIERS.stream()                    .map(carrier -> CompletableFuture                            .supplyAsync(() -> fetch(carrier, orderId), carrierExecutor)                            .orTimeout(1, TimeUnit.SECONDS)                            .exceptionally(failure -> null))                    .toList();        } catch (TaskRejectedException overloaded) {            return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)                    .header(HttpHeaders.RETRY_AFTER, "1")                    .build();        }        List<Quote> quotes = pending.stream()                .map(CompletableFuture::join)          // at most 1 s per future because of orTimeout                .filter(Objects::nonNull)                .toList();        return quotes.isEmpty()                ? ResponseEntity.status(HttpStatus.BAD_GATEWAY).build()                : ResponseEntity.ok(quotes);    }    private Quote fetch(String carrier, String orderId) {        return carriers.get()                .uri("/{carrier}/quotes/{orderId}", carrier, orderId)                .retrieve()                .body(Quote.class);    }}

Why it is written this way

  • The read timeout of 800 ms is the most important line. It caps how long one call can hold a pool thread, which caps how many threads a slow carrier can occupy: 40 calls per second × 0.8 s = 32, more than the pool, so the queue and the rejection still matter.
  • 24 threads with a queue of 48: at the normal 150 ms per call the pool drains 160 calls per second, so a full queue adds about 48 / 160 = 0.3 s of wait at worst.
  • A full queue makes supplyAsync throw TaskRejectedException in the request thread. The endpoint answers 503 with Retry-After at once instead of holding a request thread.
  • orTimeout(1, SECONDS) bounds the wait of the request thread, including time spent in the queue. exceptionally turns a failed or late carrier into a missing quote, so two healthy carriers still produce an answer.
  • If the second or third supplyAsync is rejected, the earlier ones were already accepted and still run. Their results are discarded. That waste is bounded by the pool and the read timeout.
  • JdkClientHttpRequestFactory and RestClient need Spring Framework 6.1 or later (Spring Boot 3.2 or later).

Trade-offs

  • Some requests now fail with 503 during overload. That is deliberate: the alternative was all requests failing slowly. Clients need to handle 503 and Retry-After.
  • Timeouts turn a slow success into a failure. A carrier that sometimes needs 900 ms now loses those quotes; the timeout has to be set from the carrier's real latency distribution, not guessed.
  • One slow carrier can still fill the shared pool and cause rejections for the two healthy ones. Full isolation needs a pool or a concurrency limit per carrier, which is more threads and more configuration.
  • A timed-out future does not stop the HTTP call. The pool thread stays busy until the read timeout fires, so the two limits must be chosen together.
  • The pool size, queue size and timeouts are now numbers somebody has to revisit when traffic or the carriers change. Alert on rejections and on queue length so that the numbers are checked by data.

Configuration

ThreadPoolExecutor has no flags: its settings are constructor arguments. The Spring Boot properties below are from the Spring Boot 3.5 reference. Where the reference lists no default, the value in effect comes from ThreadPoolTaskExecutor.

spring.task.execution.pool.core-sizeDefault 8
What it does
Core threads of the auto-configured applicationTaskExecutor, which runs @Async methods and Spring MVC async requests when you define no executor of your own.
When to change it
Size it from the blocking work it runs. With the default unbounded queue this is also the maximum, so it is the pool's whole capacity.
spring.task.execution.pool.queue-capacityDefault none listed (unbounded: ThreadPoolTaskExecutor uses Integer.MAX_VALUE)
What it does
Capacity of the queue. The reference notes that an unbounded capacity does not increase the pool and therefore ignores max-size.
When to change it
Set it. Choose the capacity from the wait you accept: capacity × task time ÷ threads.
spring.task.execution.pool.max-sizeDefault none listed (ThreadPoolTaskExecutor uses Integer.MAX_VALUE)
What it does
Maximum threads. The pool expands to it only when tasks are filling the queue. Ignored if the queue is unbounded.
When to change it
Has an effect only together with queue-capacity. Setting it equal to core-size gives a fixed pool that is easier to reason about.
spring.task.execution.pool.keep-aliveDefault 60s
What it does
Time a thread may stay idle before it is terminated.
When to change it
Lower it if a burst leaves many idle threads behind. It rarely matters for a fixed pool.
spring.task.execution.pool.allow-core-thread-timeoutDefault true
What it does
Whether core threads may also time out, so an idle pool shrinks to zero threads.
When to change it
Leave it on unless the first requests after a quiet period must not pay for thread creation.
spring.task.execution.thread-name-prefixDefault task-
What it does
Prefix of the names of the executor's threads.
When to change it
Change it to something that identifies the application's pool in thread dumps and logs.
spring.task.execution.shutdown.await-terminationDefault false
What it does
Whether the executor waits for tasks to complete on shutdown. await-termination-period sets the longest wait and has no default.
When to change it
Turn it on with a period shorter than the platform's termination grace period, or accepted tasks are abandoned on every deploy.
server.tomcat.threads.maxDefault 200
What it does
Maximum number of Tomcat worker threads, which is the number of requests handled at once by blocking controllers. server.tomcat.threads.min-spare (default 10) is the minimum kept alive.
When to change it
Raise it only when request threads are busy with useful waiting and the dependencies behind them can take more. Raising it during starvation makes more requests wait.
server.tomcat.accept-countDefault 100
What it does
Maximum queue length for incoming connection requests when all request processing threads are in use. server.tomcat.max-connections (default 8192) limits connections accepted and processed at one time.
When to change it
Time spent in this queue is invisible to the application's own request timer. Keep it small enough that clients fail fast rather than wait behind a stuck service.
spring.threads.virtual.enabledDefault false
What it does
On Java 21 or later, runs request handling and the auto-configured task executor on virtual threads. The pool properties above then have no effect.
When to change it
It removes the thread limit, not the limit of what you call. See the virtual threads lesson before turning it on.

Hands-on exercise

Three answers to the same overload

Drive a pool of 4 threads at five times its capacity and compare what an unbounded queue, AbortPolicy and CallerRunsPolicy each do with the excess, then size the pool so that nothing queues.

Steps · 0 of 7 done

Pool of 4, queue of 8, 100 requests per second

Simulation · not a real JVM

Change one control at a time: the queue bound, the rejection policy, the thread count and the task time. Watch active threads, queue length, queue wait and rejections.

Take it further in the simulation: Thread pool exhaustion: four threads, five times the load

What can go wrong

Setting a maximum pool size with an unbounded queue
Symptom
The pool is configured as core 10, max 100, yet a thread dump under heavy load shows exactly 10 worker threads and latency keeps rising.
Cause
Threads above core are created only when the queue refuses a task. LinkedBlockingQueue without a capacity, and Spring's default queue capacity, never refuse.
Fix
Give the queue a capacity, or set core equal to max and treat the pool as fixed. Verify with executor.pool.size under load.
Using the default executor for everything
Symptom
A slow report export makes unrelated @Async emails and MVC async requests late. task-1 to task-8 are all in the export code.
Cause
Spring Boot's auto-configured executor has 8 threads and an unbounded queue, and every unnamed @Async method shares it.
Fix
Define a named, bounded executor per kind of work and reference it with @Async("name").
Exceptions that vanish inside submit()
Symptom
A background task stops having any effect. Nothing is logged.
Cause
submit wraps the task in a Future and stores any exception there. If nobody calls get(), nobody sees it. execute would have passed it to the thread's uncaught exception handler.
Fix
Keep and check the Future, or catch and log inside the task. For @Async void methods, log in the method or register an AsyncUncaughtExceptionHandler.
CallerRunsPolicy on a request thread
Symptom
After the pool's queue fills, the HTTP request pool fills as well, and some requests run the slow task with the wrong security or transaction context.
Cause
The rejected task runs in the thread that submitted it, with that thread's thread-locals, and that thread is blocked for the task's whole duration.
Fix
Use AbortPolicy and a fast error for request paths. Keep CallerRunsPolicy for loops and listeners where slowing the producer is the intent.
Sizing from the host's core count inside a container
Symptom
A CPU-bound pool of 32 threads in a container limited to 2 CPUs: throughput is poor and the container is throttled.
Cause
The size was taken from the machine the code was written on. The JVM sees the container's CPU limit, and extra runnable threads only add context switches.
Fix
Derive CPU-bound pool sizes from Runtime.getRuntime().availableProcessors() at start-up and log the value.
Never shutting the pool down
Symptom
The JVM does not exit after main returns, or a redeploy drops work that had been accepted.
Cause
Pool threads are non-daemon by default and stay alive waiting for tasks. Stopping the process without shutdown() and awaitTermination abandons queued tasks.
Fix
Call shutdown() then awaitTermination, or let Spring manage the bean and set setWaitForTasksToCompleteOnShutdown(true) with a termination period.

How to troubleshoot

Latency climbs while CPU stays low and the error rate is flat

Check

Micrometer gauges `executor.active`, `executor.pool.size` and `executor.queued` (tag `name` = the executor bean), for example GET /actuator/metrics/executor.queued?tag=name:carrierExecutor
How to read it
Active equal to pool size with a growing queue means the pool is saturated. executor.queue.remaining at 0 means a bounded queue is full and the next submission is rejected.
Need to know what a saturated pool's threads are doing

Check

jcmd <pid> Thread.print > dump.txt, then grep -A 12 '"carrier-exec-' dump.txt
How to read it
Workers in ThreadPoolExecutor.getTask are idle. Workers with application frames above runWorker are busy: if they all show the same downstream call, that dependency is the cause. Take a second dump 10 seconds later; the same threads in the same frames means stuck, not merely busy.
Suspect the Tomcat request pool is exhausted

Check

Micrometer gauges `tomcat.threads.busy` and `tomcat.threads.config.max` (needs server.tomcat.mbeanregistry.enabled=true), and grep -c '"http-nio-' dump.txt
How to read it
Busy equal to max means every request thread is occupied and new connections wait in the accept queue. Look at where those threads are parked before raising the limit.
Clients report timeouts but the service's own request timer looks healthy

Check

Compare client-side latency or load balancer latency with the `http.server.requests` timer
How to read it
A gap means requests wait before the application sees them, in the accept queue or for a request thread. The server-side timer starts only when a thread picks the request up.
RejectedExecutionException or TaskRejectedException in the logs

Check

Read the exception message: `[Running, pool size = 4, active threads = 4, queued tasks = 2, completed tasks = 0]`
How to read it
Running with full threads and a full queue is saturation. Shutting down or Terminated means a task was submitted after shutdown(), which is a lifecycle bug and not a capacity problem.
Heap grows together with latency

Check

jcmd <pid> GC.class_histogram | head -20
How to read it
Many LinkedBlockingQueue$Node, FutureTask or lambda instances point at an unbounded executor queue holding waiting tasks.

Key takeaways

  • A task goes to a core thread, then to the queue, then to a thread above core, then to the rejection handler, in that order.
  • With an unbounded queue the maximum pool size is never used and nothing is ever rejected. Overload shows up only as waiting.
  • Executors.newFixedThreadPool and Spring Boot's default task executor both have unbounded queues. Bound them deliberately.
  • The rejection policy decides who pays for overload: AbortPolicy the caller with an error, CallerRunsPolicy the caller with time, the discard policies the task.
  • Blocking work needs about arrival rate × task time threads. CPU-bound work needs about one thread per available processor.
  • A pool can only be as healthy as the timeouts of the calls its tasks make. Bound the call before tuning the pool.
  • In a dump, idle workers are in getTask. A starved pool has every worker in the same downstream frame, often reported as RUNNABLE.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 4

A pool is created with corePoolSize 10, maximumPoolSize 50 and new LinkedBlockingQueue<>() as its queue. During a traffic peak the queue holds 4,000 tasks and p99 latency is 30 seconds. A thread dump shows exactly 10 worker threads, all busy.

Why did the pool not grow towards 50 threads?

Scenario 2 of 4

A Kafka listener thread hands each record to a pool of 8 threads with a queue of 100 for a database write of about 50 ms. During a database slowdown the queue fills. Every record must be written, and the team is happy for consumption to slow down while the database is slow.

Which configuration fits these requirements?

Scenario 3 of 4

p99 latency of a checkout API went from 300 ms to 9 s. CPU is at 6 percent. A thread dump shows 200 http-nio-8080-exec threads parked in CompletableFuture.join under CheckoutController, and all 16 payment-exec threads RUNNABLE in SocketDispatcher.read0 under PaymentClient.authorize.

What should be changed first?

Scenario 4 of 4

An image service resizes uploads, which is pure CPU work of about 80 ms per image, on a pool of 64 threads. It runs in a container limited to 4 CPUs. At peak, throughput is about 45 images per second and the container reports CPU throttling.

Which change is most likely to help?

References

References: Java SE 21 API: ThreadPoolExecutor (core and maximum pool sizes, queuing, rejected tasks) ↗ · Java SE 21 API: Executors (newFixedThreadPool, newCachedThreadPool) ↗ · Java SE 21 tool reference: jcmd (Thread.print, GC.class_histogram) ↗ · Spring Boot 3.5 reference: Task Execution and Scheduling ↗ · Spring Boot 3.5: Common Application Properties (spring.task.execution.*, server.tomcat.*) ↗ · Spring Boot 3.5 reference: Metrics (task execution and Tomcat metrics) ↗ · Spring Framework 6.2 reference: Task Execution and Scheduling (ThreadPoolTaskExecutor, @Async) ↗ · Spring Framework 6.2 API: ThreadPoolTaskExecutor ↗ · Report a problem