Load balancer

What it is

A load balancer is a server that receives requests from clients and spreads them across a pool of other servers that all do the same job, so that no single machine carries the whole load and the loss of one does not take the service with it. Configuration files and diagrams often shorten it to LB, and the work it does is called load balancing. Clients see one address, the balancer’s. The servers behind it form a pool, which nginx calls an upstream group and HAProxy calls a backend.

Every balancer makes two decisions for each request: which servers are fit to receive it, and which of those gets it. The second is the balancing method. Round robin hands requests to the servers in turn and is what nginx does when nothing else is configured. Least connections sends the next request to the server with the fewest active connections, which evens out the load when some requests take much longer than others; the HAProxy manual recommends it for long sessions such as LDAP or SQL and calls it a poor fit for short ones such as HTTP. Hashing turns a key, such as the client’s IP address or the requested URL, into a choice of server, so the same key keeps landing on the same machine. nginx also offers a random method that can pick two servers at random and send the request to the less busy one. Weights tilt the choice toward bigger machines: in the nginx guide’s own example, one server with weight=3 and two at the default weight of 1 receive three, one and one of every five requests.

Balancers also differ in how much of the traffic they read. HAProxy’s mode tcp, its default, joins client and server in a full-duplex connection and performs no layer 7 examination at all, which suits TLS passed straight through, SSH or SMTP. In mode http it parses every request before choosing a server, rejects requests that break the protocol, and can route by path or header. A balancer working at the HTTP level ends the client’s connection and opens its own to the server, so the application sees the balancer as its only client unless the balancer passes the original address along, usually in an X-Forwarded-For header. That is also why a load balancer and a reverse proxy are so often the same process: nginx is both, and its own guide describes load balancing as part of its reverse proxy implementation.

Why it matters

One server has a ceiling. When a service outgrows it, a balancer is what lets a team add a second and a third copy instead of buying a bigger machine, and every copy behind it adds capacity without the clients knowing the pool changed. The same arrangement makes maintenance ordinary: take one server out of rotation, update it, check it, put it back, and repeat for the next, while users keep being served by the rest.

It also turns the failure of a server from an outage into a smaller event. When a server stops answering, the balancer stops choosing it, and the remaining servers absorb its share. That only works if they have the headroom to do so. Three servers each running at 80% of their capacity cannot cover for one another; losing one puts 120% of a server’s work on each of the other two.

A balancer changes how the application has to be written. With round robin or least connections, nginx’s guide points out, consecutive requests from one client can reach different servers. Anything a server keeps in its own memory or on its own disk between requests, such as a login session, an uploaded file waiting for processing or a cache of the user’s basket, will be missing whenever the next request lands on another server. The application either keeps that state somewhere every server can reach, a database or a shared cache, or the balancer pins each client to one server, which nginx’s ip_hash does except when that server is unavailable.

Last, the balancer itself becomes the one machine every request passes through. A pool of ten application servers behind a single balancer has a single point of failure; it has just moved. A serious setup runs the balancer in a pair with an address that can move between them, or uses a managed balancer whose provider runs the redundancy.

What it looks like in practice

Here an nginx server spreads requests over three application servers, with a fourth held in reserve:

upstream app {
    least_conn;
    server 10.0.0.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.13:8080 max_fails=3 fail_timeout=30s weight=2;
    server 10.0.0.20:8080 backup;
}

server {
    listen 80;
    location / {
        proxy_pass http://app;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
        proxy_connect_timeout 2s;
    }
}

least_conn sends each new request to the server with the fewest active connections, and breaks ties by weighted round robin. The third server is the larger machine, so weight=2 counts it double. max_fails=3 fail_timeout=30s says that three failed attempts within 30 seconds take a server out of the pool for 30 seconds, after which nginx tries it again with live requests and puts it back if they succeed. Without those parameters the defaults apply: one failure within 10 seconds is enough. The backup server receives requests only while all three primaries are unavailable.

In the location, proxy_next_upstream decides when a request is handed on to another server: on a connection error, a timeout, or a 502 or 503 response. proxy_next_upstream_tries 2 caps how often that can happen for one request; the default, 0, means no limit. The two proxy_set_header lines keep the host name the client asked for, which nginx would otherwise replace with the upstream name app, and add the client’s address to X-Forwarded-For so the application can log real visitors instead of the balancer. proxy_connect_timeout 2s gives up on a server that has not accepted the connection within two seconds, where the default would wait 60.

These checks are passive: nginx learns that a server is sick only when real requests to it fail. HAProxy can also probe each server on its own schedule. option httpchk GET /healthz in a backend, with check on each server line, sends that request every two seconds by default; three failures in a row mark the server down and two successes bring it back. Without check, HAProxy runs no health checks and treats the server as always available.

Before nginx is a balancer it is a web server, and an ordinary one: it is one of the two behind the forty websites administered on Apache and Nginx between 2003 and 2014, which ran on Ubuntu Linux hosting servers and were tuned and maintained to stay reliable under real traffic. The backup line has a cousin one level up, where the servers are whole providers. Email built with failover across three providers keeps SendGrid as the primary with SMTP2GO and Azure Communication Services behind it, switches between them automatically, and turned a provider’s bad afternoon into a degraded path rather than an outage.

Where it goes wrong

A server that answers every request with a 500 stays in the pool. By default nginx counts only errors in talking to the server, timeouts and malformed response headers as failures. A server whose application has lost its database and returns 500 to everything answers quickly and politely, so the balancer keeps sending it a share of the traffic. The 500, 502, 503, 504 and 429 responses count against a server only when they are listed in proxy_next_upstream, and 403 and 404 never do. List the codes that mean “this server is broken”, and give the servers an active check against an endpoint that exercises what the application needs to serve a page.

One slow request runs on every server in turn. timeout is in the default proxy_next_upstream, and proxy_read_timeout defaults to 60 seconds. A report that takes 70 seconds to build times out on the first server, is handed to the second, times out there, and moves on to the third, so the client waits three minutes while every server does the same work. A timeout always counts as a failed attempt, so with the default max_fails of 1 each server it touched is also marked unavailable for 10 seconds. Cap the hand-offs with proxy_next_upstream_tries and proxy_next_upstream_timeout, give the slow endpoint its own location with a longer timeout, or move the work off the request path into a background job.

A payment goes through twice. nginx refuses by default to resend a POST, LOCK or PATCH request to another server once it has reached one, because the first server may already have acted on it. Adding non_idempotent to proxy_next_upstream to make timeouts disappear removes that guard: a payment that the first server completed but answered too slowly is completed again by the second. Leave the option off unless every endpoint behind it is idempotent, for example because it takes an idempotency key and refuses to repeat a request it has seen.

Every visitor appears to come from the balancer. Without the headers in the example, the application logs the balancer’s address for every request, and any limit kept per IP address now applies to all users at once. Adding X-Forwarded-For brings its own trap. $proxy_add_x_forwarded_for appends the client’s address to whatever X-Forwarded-For the client already sent, so the leftmost entries are whatever the client chose to write. The application should trust only the entry added by its own balancer, and accept the header only from the balancer’s address.

One office ends up on one server. nginx’s ip_hash hashes the first three octets of an IPv4 address, so every client sharing them lands on the same server, and an entire company behind one NAT gateway arrives as a single client. Changing the pool causes the opposite surprise. With a plain hash, adding or removing a server can move most keys to a different server, and every user pinned by it loses whatever their old server held. The nginx documentation advises marking a server down instead of deleting it from an ip_hash group, and offers hash ... consistent, which moves only a few keys when the pool changes. Better still, keep session state off the servers, and pinning stops mattering.

Further reading

• Using nginx as HTTP load balancer — the three basic methods, weights, session persistence and the passive health checks, each with a short configuration.
• Module ngx_http_upstream_module — every parameter of a server line, including max_fails, backup and the single-server exception, and the hash, ip_hash, least_conn and random methods.
• Module ngx_http_proxy_module — proxy_next_upstream and what counts as a failed attempt, the timeouts, and the headers nginx sets on the way to the server.
• HAProxy 3.4 Configuration Manual — balance and its algorithms, the TCP and HTTP modes, and active health checks with check, inter, rise and fall.