Thread Pool Visualization

The idea: reuse threads, queue the work

Starting a thread costs time and memory (a stack of about 1 MB, a kernel thread, scheduling). A server that started one thread per request would spend much of its time starting threads, and a burst of requests could start thousands. A thread pool keeps a few threads alive and hands them tasks (Runnables) through a work queue. A thread that finishes a task takes the next one.

The page models Java's ThreadPoolExecutor, the class behind most Java pools:

ExecutorService pool = new ThreadPoolExecutor(
        2,                                  // corePoolSize
        4,                                  // maximumPoolSize
        3, TimeUnit.SECONDS,                // keepAliveTime (3 ticks here)
        new ArrayBlockingQueue<>(3),        // bounded work queue
        new ThreadPoolExecutor.AbortPolicy());  // or CallerRunsPolicy
pool.execute(task);

Time is a clock of ticks, and every task runs for 4 ticks. The main thread submits tasks #1, #2, … Pick how many with tasks and press Submit, let time pass with Wait 1 Tick or Run Until Idle, or run a demo.

The four rules of execute()

1. Fewer than corePoolSize threads: start a new thread with the task as its first task. This happens even when another thread is idle. In Demo: light load, core threads first, #2 gets a new thread T2 although T1 has nothing to do.
2. Otherwise, offer the task to the queue. If there is room it waits there, and an idle thread takes it at once. Demo: light load shows T1, idle the longest, taking #3 from the queue.
3. Queue full, fewer than maximumPoolSize threads: start an extra thread for this task.
4. Queue full and pool at its maximum: call the RejectedExecutionHandler.

A thread runs its tasks in a loop: when one ends it takes the queue head, and with an empty queue it waits. The counters under the queue are the ones the class reports: getPoolSize(), getActiveCount(), getQueue().size(), getLargestPoolSize() and getCompletedTaskCount().

The queue fills before the pool grows

Many people expect the pool to grow first and queue later. It is the other way round. In Demo: burst of 5 fills the queue, T1 and T2 run #1 and #2, and #3 to #5 wait, although the pool could have 4 threads. The pool stays at 2 and the last task finishes at t = 12.

Only when the queue is full does the pool grow. In Demo: burst of 7 grows the pool to 4, #6 and #7 find the queue full and get new threads T3 and T4. They start at once, ahead of #3 to #5, which arrived earlier and are still queued. So a thread pool is not first-in, first-out.

This rule has a trap. With an unbounded queue (new LinkedBlockingQueue<>(), used by Executors.newFixedThreadPool) the offer never fails, so the pool never grows past corePoolSize and maximumPoolSize does nothing. The queue can grow until memory runs out.

Rejection: AbortPolicy and CallerRunsPolicy

With 4 threads busy and 3 tasks queued, the pool can take nothing more. Pick a handler and run Demo: burst of 9, rejection:

Handler	What happens to the rejected task
AbortPolicy (the default)	execute() throws RejectedExecutionException; the caller must handle it. #8 and #9 are lost.
CallerRunsPolicy	The thread that called execute() runs the task itself. main spends 4 ticks on #8, and meanwhile the pool works off its queue. When main comes back, #9 fits in the queue.
DiscardPolicy	The task is dropped silently (not on the page).
DiscardOldestPolicy	The oldest queued task is dropped and the new one is offered again (not on the page).

CallerRunsPolicy is a simple form of back-pressure. A producer that submits too fast is made to do the work itself, so it slows down to the speed of the pool, and no task is lost.

keepAliveTime: shrinking back

A thread that finds the queue empty waits for a task. If the pool has more than corePoolSize threads it waits in poll(keepAliveTime); when that times out with no task and the pool is still above the core size, the thread exits. At or below the core size it waits in take(), which never times out.

In Demo: idle threads time out, T4 goes idle at t = 4 and exits at t = 7. T1, T2 and T3 go idle at t = 8 and all time out at t = 11. T1 exits first; then the pool is down to 2, so T2 and T3 stay and switch to take(). The pool does not remember which threads were "core": corePoolSize is a count, not a set of threads. allowCoreThreadTimeOut(true) lets the last core threads time out too.

Executors and pool sizes

The factory methods in Executors are this one class with different settings:

Factory	core / max	Queue
newFixedThreadPool(n)	n / n	unbounded LinkedBlockingQueue
newCachedThreadPool()	0 / unlimited, keepAlive 60 s	SynchronousQueue (no room: every task needs a free thread)
newSingleThreadExecutor()	1 / 1	unbounded

How many threads? For work that only uses the CPU, about one per core: more threads just take turns. For work that waits on I/O, more: roughly cores × (1 + wait time / compute time). A bounded queue plus a rejection policy is what keeps a slow pool from taking the whole application down with it.

See also Java ForkJoinPool and Work Stealing for a pool where every thread has its own queue, Database Connection Pool for pooling connections instead of threads, and Bounded Buffer for how a blocking queue like ArrayBlockingQueue works inside.

With virtual threads you stop pooling threads for blocking work; see Java Virtual Threads vs Platform Threads.

What the page leaves out

Real tasks take different times, and threads finishing at the same moment race; the page settles every race in thread order. It also leaves out submit() and Futures, shutdown() and shutdownNow(), the ThreadFactory that names and creates threads, prestartAllCoreThreads(), ScheduledThreadPoolExecutor, exceptions thrown by tasks (they end the worker, which is replaced), and the single atomic ctl word in which the real class keeps both the run state and the worker count.

References

Java SE 21 API: ThreadPoolExecutor

Java SE 21 API: Executors

OpenJDK source: ThreadPoolExecutor.java (execute, getTask)

The Java Tutorials: Thread Pools