The idea: one address in front, many servers behind
One server can only handle so many requests, and when it crashes the site is gone. So a site runs several copies of its application (the backends) and puts a load balancer in front: clients connect to one address, and the balancer decides which backend serves them. The big question is what it decides on. A layer 4 balancer works with TCP connections. A layer 7 balancer works with HTTP requests.
On the canvas, the clients are on the left: Alice, Bob, and three office users who share one public IP address behind their
office NAT. The balancer is in the middle. It shows its configuration, what it can see of the traffic, its last decision, the health
checks, and either its connection table (L4) or its own pool of connections to the backends (L7). The backends are on the right:
three copies of the API and one static file server. Each backend serves one request at a time and queues the rest,
so you can see work pile up. Every Send takes 1 second; /api/users takes 2 s to serve and /report takes 9 s.
Layer 4: forwarding connections
When a client opens a TCP connection, an L4 balancer (HAProxy mode tcp, nginx stream {}, Linux IPVS,
AWS Network Load Balancer) picks a backend right away, before a single request byte has arrived, and records the
choice in a connection table. From then on it rewrites the destination address of every packet of that connection (DNAT) and
passes the bytes through. Because the traffic is TLS, the bytes are encrypted: the balancer sees addresses and ports, nothing more.
Every request on that connection goes to the same backend (Demo 1).
This is cheap and works for any TCP protocol: databases, SMTP, MQTT, not only HTTP. Some L4 balancers even let the backend answer the client directly, bypassing the balancer on the way back (direct server return), which helps when responses are much bigger than requests.
Layer 7: reading every request
An L7 balancer (HAProxy mode http, nginx http { upstream }, Envoy, AWS Application Load Balancer) ends
the client's TCP connection and its TLS session itself, so the certificate lives on the balancer. Now it can read each
request: method, path, Host, cookies. It picks a backend per request and sends it over its own
keep-alive connections, which it reuses for later requests (Demo 2). It adds headers such as
X-Forwarded-For, so the backend still knows the client's address.
Reading the request makes several things possible that L4 cannot do:
- Routing by path or host:
/static/goes to the static pool (Demo 3). At L4 the request lands on an API server, which answers 404. - Spreading one HTTP/2 connection: a browser sends many requests as streams on one connection. At L4 they all land on one backend; at L7 each stream is balanced on its own (Demo 4). The same applies to gRPC.
- Retrying a request on another backend when one fails (Demo 7).
- Sticky cookies (Demo 8), header rewriting, compression, caching, rate limiting.
The price is more work per request (TLS, HTTP parsing, two connections instead of one) and the private key on the balancer.
Choosing a backend
| Algorithm | Picks | Config |
|---|---|---|
| round robin | the next backend in turn; weights give bigger servers more turns | balance roundrobin, nginx default |
| least connections | the backend with the fewest open connections (or requests in flight) | balance leastconn, least_conn |
| source-IP hash | hash(client IP) mod N: the same client always lands on the same backend | balance source, ip_hash |
| consistent hash | a point on a hash ring; adding a backend moves only a few keys (see Consistent Hashing) | hash $request_uri consistent |
| power of two choices | two random backends, the less loaded of them | random two least_conn |
Round robin counts turns, not work. If some requests are slow, they can all land on the same backend, and the fast requests queued
behind them wait. Least connections looks at what each backend is still doing. In Demo 5, the same nine requests are sent
twice: with round robin all three slow /report requests end up on api-1 and the last one waits 21 s; with
least connections the slowest response takes 9 s.
Health checks and failures
The balancer checks every backend regularly (here every 3 s, HAProxy check inter 3s fall 2 rise 2). A backend that
fails two checks in a row is marked DOWN and gets no new traffic; after two good checks it is UP
again. Between the crash and the second failed check, the balancer still sends it traffic. nginx open source does
passive checks instead: max_fails=1 fail_timeout=10s takes a server out for 10 s after a failed request.
What the client sees depends on the layer. At L4, every connection to the crashed backend is reset, and a new connection sent there
before it is marked DOWN is refused (Demo 6). At L7, the balancer still holds the request, so it sends it to another backend
(proxy_next_upstream error, HAProxy retries and option redispatch); the client only sees a slower
answer (Demo 7). Retrying is only safe for requests that can be repeated: a GET, not a payment. See
Circuit Breaker for the same problem seen from the caller.
Sticky sessions
If a backend keeps the user's session in its own memory (a shopping cart, a login), every request of that user must go back to it.
An L7 balancer can insert a cookie such as SERVERID=api-2 in the first response and follow it afterwards (Demo 8).
An L4 balancer can only hash the source IP. That breaks in two ways. When the backend dies, the session is gone anyway. And many users
behind one NAT address all hash to the same backend (Demo 9). The better fix is a stateless backend: keep sessions in a shared
store such as Redis, so any backend can serve any request.
L4 and L7 side by side
| L4 (TCP) | L7 (HTTP) | |
|---|---|---|
| decides per | connection | request |
| sees | IP addresses and ports | method, path, host, headers, cookies |
| TLS | passes through, encrypted | ends at the balancer (certificate there) |
| connections | one, client to backend | two: client to balancer, balancer to backend (pooled) |
| protocols | any TCP (or UDP) | HTTP, HTTP/2, gRPC, WebSocket |
| on failure | connections reset | request retried |
| cost | low | higher (TLS, parsing) |
| examples | IPVS, AWS NLB, HAProxy mode tcp, nginx stream | nginx, HAProxy mode http, Envoy, AWS ALB |
Large sites often use both: an L4 tier spreads connections over many L7 balancers, which then route the requests. See How nginx Handles a Request for what one L7 balancer does inside, and Kubernetes Service Types for the cloud load balancer in front of a cluster. Canary and Blue/Green Deployments shows weighted target groups moving traffic to a new version. How a CDN and Route 53 Work shows the layer in front of the balancers: DNS routing between regions and edge caches.
What the page leaves out
- DNS, anycast and global load balancing in front of the balancers, and keeping the balancer itself highly available (VRRP, keepalived).
- Weights, slow start for a backend that just came back, and connection draining before a backend is removed.
- Real backends serve many requests at once; here each has a single worker so queues are easy to see.
- How kernel balancers such as IPVS or Google's Maglev forward packets, and client-side balancing in service meshes.
- WebSockets and other long-lived connections, which stay on one backend at L7 too.