One thread per connection does not scale
The most common mental model of a server is simple: a connection arrives, a thread opens, it is served, it closes. That works with a hundred clients. With ten thousand, the system crawls. And not for lack of CPU, but for something far more mundane: every thread needs its own stack (1 to 8 MB reserved by default), and the kernel scheduler has to rotate between thousands of tasks that spend most of their time asleep, waiting for a byte to arrive on a socket. That scenario earned its own name two decades ago: the C10K problem, serving ten thousand simultaneous connections without the server drowning.
The way out was not more memory or more cores, but changing the idea: instead of one keeper per connection, let the kernel watch all of them at once and wake the process only when one of them has something to say. That is I/O multiplexing, and its centrepiece on Linux is called epoll.
From blind polling to events
The first generation was select(), and its limit is both historical and painful: the process hands the kernel a bitmask with the file descriptors (the integers identifying each open socket) it cares about, and the kernel walks that entire list to see which ones are ready. On top of that, the mask is fixed in size: FD_SETSIZE, traditionally 1024. If you have 900 descriptors watching a single new connection, the cost is the same.
poll() removed the 1024 limit by passing an array of structures instead of a bitmask, but kept the underlying problem: on every call, the kernel checks all descriptors one by one, even if only one of them changed. The cost grows linearly with the number of watched connections, so going from 1,000 to 10,000 clients multiplies by ten the work done for every useful event.
Three calls and two data structures
epoll changes the architecture. Instead of sending the descriptor list on every query, the process creates a persistent instance inside the kernel with epoll_create(). That instance keeps two structures that explain almost all of its performance:
- A red-black tree (a self-balancing binary tree with logarithmic lookups) holding every registered descriptor. Entries are added and removed with
epoll_ctl(), called once per connection, not on every iteration. - A ready list, a linked queue containing only the descriptors with pending activity.
When the process calls epoll_wait(), the kernel walks nothing: it returns the elements of the ready list directly. The cost depends on how many events there are, not on how many sockets you are watching. That is the leap: going from 1,000 to 10,000 connections stops making the operation more expensive.
Who fills that ready list? Not the process, but the kernel itself. When the network driver receives packets, it processes them in softirqs (deferred interrupt routines, the part of network work performed outside hard interrupt context) and pushes the data up the TCP stack. On reaching the socket, the kernel adds that descriptor to the ready list of the epoll instance where it is registered and, if someone was blocked in epoll_wait(), wakes them up. The process does not ask: it gets told.
Level versus edge: the detail that breaks servers
Here is the nuance that separates a correct implementation from one that loses data. epoll works by default in level-triggered mode: if it tells you there are bytes to read and you consume only part of them, it will tell you again on the next call. It is forgiving and tolerates mistakes.
In edge-triggered mode (flag EPOLLET) it only warns you when the state changes: the transition from “no data” to “data available”. If you do not read until the socket buffer is drained, there will be no second notification and the connection will stay mute forever. High-performance servers use this mode because it avoids repeated wake-ups, but it forces you to read in a loop until the kernel returns EAGAIN, that is, until nothing is left. Nginx, Redis or Node.js live on exactly this mechanics: an event loop (the reactor pattern) that asks the kernel what is ready, dispatches the matching task and waits again.
What else you save
The benefit is not only CPU. With ten thousand threads, the scheduler burns time on context switches —saving registers, swapping stacks, invalidating CPU caches— and memory fills up with almost always empty stacks. With epoll, a single thread manages every socket and live connections take only a few structures per socket: send and receive buffers, TCP sequence numbers, timers. Memory grows with data in flight, not with the number of open conversations.
The trade-off is that work must be short and non-blocking: one slow database query on the event thread halts all ten thousand clients at once. That is why these servers delegate heavy lifting to a few worker threads or separate processes.
The next step: io_uring
epoll is still a notification interface: first you ask, then you read. io_uring, available since Linux 5.1, flips the scheme into a completion model: the process drops operations into a submission queue shared with the kernel (two ring buffers mapped with mmap, one for requests and one for results) and collects outcomes from the completion queue, often without a single system call per operation. It is the same underlying idea —do not block— pushed into disk reads, networking and nearly any operation.
Two decades after C10K, the lesson stands: when the bottleneck is not the machine but the way you talk to the kernel, the fix is usually not more resources, but a different conversation.
