{"canonical_url":"https://engineer.company/glossary/load-balancer/","content_markdown":"# Load balancer\n\n## What it is\n\nA load balancer is a server that receives requests from clients and spreads them\nacross a pool of other servers that all do the same job, so that no single\nmachine carries the whole load and the loss of one does not take the service\nwith it. Configuration files and diagrams often shorten it to LB, and the work\nit does is called load balancing. Clients see one address, the balancer's. The\nservers behind it form a pool, which nginx calls an upstream group and HAProxy\ncalls a backend.\n\nEvery balancer makes two decisions for each request: which servers are fit to\nreceive it, and which of those gets it. The second is the balancing method.\nRound robin hands requests to the servers in turn and is what nginx does when\nnothing else is configured. Least connections sends the next request to the\nserver with the fewest active connections, which evens out the load when some\nrequests take much longer than others; the HAProxy manual recommends it for long\nsessions such as LDAP or SQL and calls it a poor fit for short ones such as\nHTTP. Hashing turns a key, such as the client's IP address or the requested URL,\ninto a choice of server, so the same key keeps landing on the same machine.\nnginx also offers a random method that can pick two servers at random and send\nthe request to the less busy one. Weights tilt the choice toward bigger\nmachines: in the nginx guide's own example, one server with `weight=3` and two\nat the default weight of 1 receive three, one and one of every five requests.\n\nBalancers also differ in how much of the traffic they read. HAProxy's\n`mode tcp`, its default, joins client and server in a full-duplex connection and\nperforms no layer 7 examination at all, which suits TLS passed straight through,\nSSH or SMTP. In `mode http` it parses every request before choosing a server,\nrejects requests that break the protocol, and can route by path or header. A\nbalancer working at the HTTP level ends the client's connection and opens its\nown to the server, so the application sees the balancer as its only client\nunless the balancer passes the original address along, usually in an\n`X-Forwarded-For` header. That is also why a load balancer and a reverse proxy\nare so often the same process: nginx is both, and its own guide describes load\nbalancing as part of its reverse proxy implementation.\n\n## Why it matters\n\nOne server has a ceiling. When a service outgrows it, a balancer is what lets a\nteam add a second and a third copy instead of buying a bigger machine, and every\ncopy behind it adds capacity without the clients knowing the pool changed. The\nsame arrangement makes maintenance ordinary: take one server out of rotation,\nupdate it, check it, put it back, and repeat for the next, while users keep\nbeing served by the rest.\n\nIt also turns the failure of a server from an outage into a smaller event. When\na server stops answering, the balancer stops choosing it, and the remaining\nservers absorb its share. That only works if they have the headroom to do so.\nThree servers each running at 80% of their capacity cannot cover for one\nanother; losing one puts 120% of a server's work on each of the other two.\n\nA balancer changes how the application has to be written. With round robin or\nleast connections, nginx's guide points out, consecutive requests from one\nclient can reach different servers. Anything a server keeps in its own memory or\non its own disk between requests, such as a login session, an uploaded file\nwaiting for processing or a cache of the user's basket, will be missing whenever\nthe next request lands on another server. The application either keeps that\nstate somewhere every server can reach, a database or a shared cache, or the\nbalancer pins each client to one server, which nginx's `ip_hash` does except\nwhen that server is unavailable.\n\nLast, the balancer itself becomes the one machine every request passes through.\nA pool of ten application servers behind a single balancer has a single point of\nfailure; it has just moved. A serious setup runs the balancer in a pair with an\naddress that can move between them, or uses a managed balancer whose provider\nruns the redundancy.\n\n## What it looks like in practice\n\nHere an nginx server spreads requests over three application servers, with a\nfourth held in reserve:\n\n```nginx\nupstream app {\n    least_conn;\n    server 10.0.0.11:8080 max_fails=3 fail_timeout=30s;\n    server 10.0.0.12:8080 max_fails=3 fail_timeout=30s;\n    server 10.0.0.13:8080 max_fails=3 fail_timeout=30s weight=2;\n    server 10.0.0.20:8080 backup;\n}\n\nserver {\n    listen 80;\n    location / {\n        proxy_pass http://app;\n        proxy_set_header Host $host;\n        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n        proxy_next_upstream error timeout http_502 http_503;\n        proxy_next_upstream_tries 2;\n        proxy_connect_timeout 2s;\n    }\n}\n```\n\n`least_conn` sends each new request to the server with the fewest active\nconnections, and breaks ties by weighted round robin. The third server is the\nlarger machine, so `weight=2` counts it double. `max_fails=3 fail_timeout=30s`\nsays that three failed attempts within 30 seconds take a server out of the pool\nfor 30 seconds, after which nginx tries it again with live requests and puts it\nback if they succeed. Without those parameters the defaults apply: one failure\nwithin 10 seconds is enough. The `backup` server receives requests only while\nall three primaries are unavailable.\n\nIn the `location`, `proxy_next_upstream` decides when a request is handed on to\nanother server: on a connection error, a timeout, or a 502 or 503 response.\n`proxy_next_upstream_tries 2` caps how often that can happen for one request;\nthe default, 0, means no limit. The two `proxy_set_header` lines keep the host\nname the client asked for, which nginx would otherwise replace with the upstream\nname `app`, and add the client's address to `X-Forwarded-For` so the application\ncan log real visitors instead of the balancer. `proxy_connect_timeout 2s` gives\nup on a server that has not accepted the connection within two seconds, where\nthe default would wait 60.\n\nThese checks are passive: nginx learns that a server is sick only when real\nrequests to it fail. HAProxy can also probe each server on its own schedule.\n`option httpchk GET /healthz` in a backend, with `check` on each `server` line,\nsends that request every two seconds by default; three failures in a row mark\nthe server down and two successes bring it back. Without `check`, HAProxy runs\nno health checks and treats the server as always available.\n\nBefore nginx is a balancer it is a web server, and an ordinary one: it is one of\nthe two behind the\n[forty websites administered on Apache and Nginx](/portfolio/administered-40-websites-on-apache-and-nginx/)\nbetween 2003 and 2014, which ran on Ubuntu Linux hosting servers and were tuned\nand maintained to stay reliable under real traffic. The `backup` line has a\ncousin one level up, where the servers are whole providers.\n[Email built with failover across three providers](/portfolio/built-email-with-failover-and-delivery-logging/)\nkeeps SendGrid as the primary with SMTP2GO and Azure Communication Services\nbehind it, switches between them automatically, and turned a provider's bad\nafternoon into a degraded path rather than an outage.\n\n## Where it goes wrong\n\n**A server that answers every request with a 500 stays in the pool.** By default\nnginx counts only errors in talking to the server, timeouts and malformed\nresponse headers as failures. A server whose application has lost its database\nand returns 500 to everything answers quickly and politely, so the balancer\nkeeps sending it a share of the traffic. The 500, 502, 503, 504 and 429\nresponses count against a server only when they are listed in\n`proxy_next_upstream`, and 403 and 404 never do. List the codes that mean \"this\nserver is broken\", and give the servers an active check against an endpoint that\nexercises what the application needs to serve a page.\n\n**One slow request runs on every server in turn.** `timeout` is in the default\n`proxy_next_upstream`, and `proxy_read_timeout` defaults to 60 seconds. A report\nthat takes 70 seconds to build times out on the first server, is handed to the\nsecond, times out there, and moves on to the third, so the client waits three\nminutes while every server does the same work. A timeout always counts as a\nfailed attempt, so with the default `max_fails` of 1 each server it touched is\nalso marked unavailable for 10 seconds. Cap the hand-offs with\n`proxy_next_upstream_tries` and `proxy_next_upstream_timeout`, give the slow\nendpoint its own `location` with a longer timeout, or move the work off the\nrequest path into a background job.\n\n**A payment goes through twice.** nginx refuses by default to resend a POST,\nLOCK or PATCH request to another server once it has reached one, because the\nfirst server may already have acted on it. Adding `non_idempotent` to\n`proxy_next_upstream` to make timeouts disappear removes that guard: a payment\nthat the first server completed but answered too slowly is completed again by\nthe second. Leave the option off unless every endpoint behind it is\n[idempotent](/glossary/#idempotent), for example because it takes an idempotency\nkey and refuses to repeat a request it has seen.\n\n**Every visitor appears to come from the balancer.** Without the headers in the\nexample, the application logs the balancer's address for every request, and any\nlimit kept per IP address now applies to all users at once. Adding\n`X-Forwarded-For` brings its own trap. `$proxy_add_x_forwarded_for` appends the\nclient's address to whatever `X-Forwarded-For` the client already sent, so the\nleftmost entries are whatever the client chose to write. The application should\ntrust only the entry added by its own balancer, and accept the header only from\nthe balancer's address.\n\n**One office ends up on one server.** nginx's `ip_hash` hashes the first three\noctets of an IPv4 address, so every client sharing them lands on the same\nserver, and an entire company behind one NAT gateway arrives as a single client.\nChanging the pool causes the opposite surprise. With a plain `hash`, adding or\nremoving a server can move most keys to a different server, and every user\npinned by it loses whatever their old server held. The nginx documentation\nadvises marking a server `down` instead of deleting it from an `ip_hash` group,\nand offers `hash ... consistent`, which moves only a few keys when the pool\nchanges. Better still, keep session state off the servers, and pinning stops\nmattering.\n\n## Further reading\n\n- [Using nginx as HTTP load balancer](https://nginx.org/en/docs/http/load_balancing.html)\n  — the three basic methods, weights, session persistence and the passive health\n  checks, each with a short configuration.\n- [Module ngx_http_upstream_module](https://nginx.org/en/docs/http/ngx_http_upstream_module.html)\n  — every parameter of a `server` line, including `max_fails`, `backup` and the\n  single-server exception, and the `hash`, `ip_hash`, `least_conn` and `random`\n  methods.\n- [Module ngx_http_proxy_module](https://nginx.org/en/docs/http/ngx_http_proxy_module.html)\n  — `proxy_next_upstream` and what counts as a failed attempt, the timeouts, and\n  the headers nginx sets on the way to the server.\n- [HAProxy 3.4 Configuration Manual](https://docs.haproxy.org/3.4/configuration.html)\n  — `balance` and its algorithms, the TCP and HTTP modes, and active health\n  checks with `check`, `inter`, `rise` and `fall`.\n\n\u003chttps://engineer.company/glossary/load-balancer/\u003e\n","date_modified":"2026-10-29T11:17:41Z","description":"How a load balancer picks a server for each request, how it notices a sick one, and five ways a balanced pool still lets users down.","formats":{"gemtext":"https://engineer.company/glossary/load-balancer/index.gmi","gopher":"https://engineer.company/glossary/load-balancer/gophermap","html":"https://engineer.company/glossary/load-balancer/","json":"https://engineer.company/glossary/load-balancer/index.json","markdown":"https://engineer.company/glossary/load-balancer/index.md","pdf":"https://engineer.company/glossary/load-balancer/index.pdf"},"kind":"page","language":"en","title":"What is a load balancer?","url":"https://engineer.company/glossary/load-balancer/","version":1}