Load Balancing

Once we scale out to many servers (previous note), something has to decide which server each request goes to. That something is a load balancer. In simple language — it’s a traffic cop sitting in front of our servers, spreading incoming requests across all of them so no single box gets hammered.

What it actually does

A load balancer accepts every incoming request and forwards it to one of the backend servers (the “pool” or “upstream”). Clients only ever talk to the load balancer’s address — they have no idea there are five servers behind it, or which one served them.

That indirection buys us three things at once:

• Distribution — spread load so no server melts while others sit idle.
• Availability — if a server dies, stop sending it traffic and route around it.
• Flexibility — add or remove servers without clients noticing (great for deploys and auto-scaling).

L4 vs L7

Load balancers work at one of two layers, and the difference is how much of the request they look at.

• L4 (transport layer) balances on TCP/UDP info only — source/destination IP and port. It doesn’t read the actual request content. It just forwards packets. Because it does so little inspection, it’s blazing fast and protocol-agnostic. Think of it like a mail sorter routing by envelope address without opening the letter.
• L7 (application layer) reads the actual HTTP request — the URL path, headers, cookies. So it can route /api/* to one pool and /images/* to another, terminate TLS, and do content-aware tricks. Smarter, but it does more work per request.

The only real difference is: L4 routes by address, L7 routes by content. Most modern web setups use L7 (NGINX, HAProxy, AWS ALB) because path/header routing is so useful.

Distribution algorithms

Once a request arrives, which server gets it? A few standard strategies:

• Round robin — hand requests out in a rotating order: 1, 2, 3, 1, 2, 3… Dead simple, assumes all servers are equal.
• Least connections — send the request to whichever server currently has the fewest active connections. Better when requests take uneven amounts of time.
• IP hash — hash the client’s IP and always send that client to the same server. Gives us stickiness for free.
• Weighted — give beefier servers a bigger share (weight 3 gets three times the traffic of weight 1). Handy for mixed hardware.

upstream app_servers {
    least_conn;                 # pick the least-busy backend
    server 10.0.0.1:8080 weight=3;
    server 10.0.0.2:8080;
    server 10.0.0.3:8080;
}

server {
    location / {
        proxy_pass http://app_servers;
    }
}

Health checks

The load balancer constantly pings each backend to ask “are you alive?” — usually a GET /health that should return 200 OK. If a server fails a few checks in a row, the LB marks it unhealthy and stops routing to it until it recovers. This is what makes horizontal scaling actually reliable: a dead box just quietly drops out of rotation instead of serving errors to users.

Sticky sessions

Normally any server can handle any request — that’s the goal (stateless servers). But sometimes a server holds per-user state in memory (an old-school session). Sticky sessions (session affinity) pin a given user to the same backend for their whole visit, usually via a cookie or IP hash.

It works, but it’s a bit of a smell. It fights the whole point of load balancing — if that one server dies, those users lose their session, and load can’t spread evenly. The cleaner fix is to make servers stateless and push session data into shared storage like Redis. Then any server can serve any request, and we don’t need stickiness at all.

Interview soundbite

A load balancer fronts a pool of servers and spreads requests across them, giving us distribution, availability, and easy scaling. L4 balances on IP/port without reading the request; L7 reads the HTTP content so it can route by path or header. Common algorithms are round robin, least connections, IP hash, and weighted. Health checks pull dead servers out of rotation automatically, and while sticky sessions pin a user to one backend, we’d rather keep servers stateless and store sessions in Redis so any server can handle any request.