The idea: few threads, one kernel object

Tomcat's NIO connector can hold 10,000 open connections with about 200 threads. It can because a connection that is not sending anything needs no thread at all. Three kinds of Tomcat threads share the work, and one kernel object, an epoll instance, tells them which sockets need attention.

User space: the Acceptor thread calls accept() on the listening socket, the Poller thread sleeps in epoll_wait on the epoll instance, worker threads read, run the servlet and write. Kernel: the epoll instance holds an RB-tree of watched sockets, a ready list and a wait queue; a socket with new data goes on the ready list and wakes the Poller.
The Acceptor creates connections, the Poller waits on epoll for all of them at once, and a worker runs only when a socket has data.
  • The Acceptor thread takes finished connections from the kernel with accept().
  • The Poller thread owns a Java Selector. On Linux the JDK builds it on epoll. The Poller sleeps in epoll_wait() until some socket is ready.
  • Worker threads (exec-1, exec-2, …, at most maxThreads, default 200) read the request, run your code and write the response.

The five numbered steps on the left of the canvas light up as the animation goes. Demo: one request, end to end shows all of them.

Step 1: accept

The kernel does the TCP handshake (SYN, SYN-ACK, ACK) without Tomcat. The finished connection waits in the listening socket's accept queue; its length is the Connector's acceptCount (100). The Acceptor's accept() takes it off the queue and returns a new file descriptor. The Acceptor then calls configureBlocking(false), which is fcntl(fd, O_NONBLOCK): from now on read() and write() on this socket return at once instead of waiting.

Step 2: register with epoll

Only the Poller thread may change its Selector, so the Acceptor puts a PollerEvent in the Poller's queue and calls selector.wakeup(). The JDK wakes the Poller by writing to an eventfd that is itself watched by the epoll instance, so the wake-up uses the same path as a socket with data. The Poller runs the event (channel.register(selector, OP_READ)), and in its next select() the JDK calls:

epoll_ctl(epfd, EPOLL_CTL_ADD, 17, {EPOLLIN})

The kernel allocates an epitem for fd 17, inserts it in the instance's red-black tree (the interest list, O(log n)), and hooks a callback, ep_poll_callback, onto the socket's own wait queue.

Step 3: wait for events

A struct eventpoll has three parts, all drawn on the canvas:

PartHoldsChanged by
rbr, RB-treeOne epitem per watched fdepoll_ctl ADD / MOD / DEL
rdllist, ready listThe epitems whose fd has an eventep_poll_callback adds; epoll_wait takes
wq, wait queueThreads sleeping in epoll_waitepoll_wait sleeps; the callback wakes

When bytes arrive, the TCP/IP stack puts them in the socket's receive queue and wakes the socket's waiters. That runs ep_poll_callback, which adds the epitem to the ready list and wakes the Poller. epoll_wait then copies out only the ready list. Its cost depends on how many fds are ready, not on how many are watched (Demo: 3 connections, only 1 busy). The older select() and poll() system calls scan every fd on every call.

Time runs left to right: a packet arrives on fd 17, the receive queue fills, ep_poll_callback puts fd 17 on the ready list, the Poller is woken from the wait queue, and epoll_wait returns one event: fd 17.
From packet to Poller: the kernel moves the socket to the ready list and wakes the sleeper, so the Poller never scans idle sockets.

Step 4: dispatch to a worker

For each ready key the Poller first sets the key's interest to 0 (unreg), so the same socket is not reported again while a worker reads it. The JDK applies this at the next select() as epoll_ctl(EPOLL_CTL_DEL). Then the Poller calls executor.execute(new SocketProcessor(...)) and goes back to sleep. It never reads a byte of HTTP itself, so one Poller can serve thousands of sockets.

Step 5: process the request

The worker reads the bytes, Http11Processor parses the request, and the servlet (here Spring's DispatcherServlet) calls the controller, the service and the repository. JDBC is blocking I/O: the worker waits for the database. That is fine, because only that worker waits. The Poller and the other workers keep running (Demo: two requests at once). If all maxThreads workers wait, new requests queue; see How Tomcat handles an HTTP request for the limits.

Large responses and OP_WRITE

A non-blocking write() copies only what fits in the socket's send queue and returns 0 when the queue is full. Tomcat then registers the socket for OP_WRITE and the worker waits (a servlet's normal write is blocking from the servlet's point of view). When the client's ACKs free the send queue, the socket becomes writable, EPOLLOUT puts it on the ready list, and the Poller wakes the waiting worker, which writes the rest (Demo: large response waits for OP_WRITE).

Keep-alive and close

After the response the worker hands the socket back to the Poller for OP_READ and returns to the pool. An idle keep-alive connection now costs an fd, an epitem and some buffers, but no thread. When the client closes, its FIN makes the socket readable; the worker's read() returns -1 and Tomcat calls close() (Demo: client closes the connection). The TCP states are on the TCP socket page.

See also Linux epoll for level- and edge-triggered mode, and Nginx for a server whose worker processes run the request on the same thread that waits on epoll.

What the page leaves out

  • The real code paths: NioEndpoint uses LimitLatch for maxConnections, a wakeupCounter so only one thread calls wakeup(), selectNow() when events are waiting, and timeouts checked after each select(1000).
  • Level-triggered details: the kernel puts a reported epitem back on the ready list and re-checks it on the next epoll_wait. The page clears the list at once because the Poller removes the fd before it waits again.
  • The RB-tree is drawn balanced, without the kernel's rotations. File descriptor numbers are taken from the diagram, not from the kernel's lowest-free rule.
  • TLS, HTTP/2, async servlets and virtual threads, all of which change what a worker does.