Load Balancing: L4 vs L7, Algorithms, Health Checks

The first web-scale balancing was DNS (Domain Name System) round robin in the mid-1990s: publish several IP addresses for one name and let resolvers rotate them. It spreads load but cannot see a dead server, so hardware appliances followed (Cisco LocalDirector, F5 BIG-IP in 1997), then software: LVS (Linux Virtual Server, 1998) in the kernel, HAProxy (2001) and Nginx (2004) as user-space proxies. Cloud made it a checkbox: AWS ELB (Elastic Load Balancing) in 2009, split in 2016-2017 into ALB (Application Load Balancer, layer 7) and NLB (Network Load Balancer, layer 4). Google published Maglev, its software layer 4 balancer, in 2016, and Lyft open-sourced Envoy the same year, which became the proxy inside Istio and most service meshes.

You have met one every time you wrote an Nginx upstream block, created a Kubernetes Service (kube-proxy balances connections to its pods), or pointed a domain at an ALB target group. The smallest working example is four lines:

Every horizontally scaled service has one, and most of its failure modes are balancing failures in disguise. A deploy that drops a few hundred requests is usually a missing drain. One pod at 100% CPU while its siblings idle is round robin meeting uneven requests, or an L4 balancer meeting long-lived HTTP/2 connections. A whole fleet "going down" because the database blipped is a health check that checked too much. And every retry the balancer makes on your behalf is a decision about idempotency you may not know you took.

An L4 balancer picks a backend when a connection opens (the first SYN packet for TCP) and then forwards every packet of that connection to the same backend, rewriting the destination address with NAT (network address translation) or, in DSR (direct server return) setups like LVS-DR and Maglev, letting the backend reply to the client directly. It never decrypts anything, so it is fast and protocol agnostic, but its unit of balancing is the connection.

An L7 balancer is a reverse proxy. It completes the TLS (Transport Layer Security) handshake itself, parses HTTP, and opens its own, usually pooled, connections to the backends. Because it sees each request, it can pick a backend per request, route on path, host or header, retry, add X-Forwarded-For, and count real request latency. The cost is CPU for TLS and parsing, and that it must understand the protocol.

Round robin is the default in Nginx, Envoy and ALB because it is stateless and perfectly fair when every request costs the same. Real requests do not: one endpoint returns a cached row in 2 ms, another builds a report for 4 seconds. Round robin keeps handing work to the instance that is already stuck on reports, because it counts requests sent, not requests still running.

Picking the global minimum has its own problem. With ten balancer instances, each with a slightly stale view, all ten see the same "least loaded" backend and pile onto it at once. The fix, from Mitzenmacher's 2001 "power of two choices" result, is P2C (power of two choices): pick two backends at random and send to the less loaded of the two. It needs no global scan, avoids the herd, and in the balls-into-bins model cuts the worst backend's excess load exponentially compared with one random choice. Envoy's LEAST_REQUEST and Nginx's random two least_conn (since 1.15.1) both do exactly this.

A balancer is only as good as its list of healthy backends. Active checks probe each backend on a timer (ALB defaults to every 30 seconds and marks a target unhealthy after 2 failures, healthy after 5 successes). Passive checks watch real traffic: open-source Nginx's max_fails/fail_timeout and Envoy's outlier detection eject a backend after consecutive errors. Active checks catch a backend before users hit it; passive checks react within a few requests instead of a few intervals. Production setups use both.

Health checks are also how deploys avoid dropping requests. The balancer must learn that an instance is leaving before the instance stops accepting work, and must not send to a new instance before it is ready. The sequence for one instance in a rolling deploy:

"The load balancer" in a large system is a stack of them, each spreading load across the next tier and each redundant in its own way. The top tier is not a box at all: DNS returns several addresses, and anycast plus ECMP (equal-cost multi-path routing) lets routers spread packets for one IP over many identical L4 balancers. That is the real answer to "is the balancer a single point of failure?"; the smaller answer is an active-passive pair sharing a VIP through VRRP (Virtual Router Redundancy Protocol), for example with keepalived.

gRPC runs over HTTP/2, which multiplexes every call onto one long-lived connection. Behind an L4 balancer, including a plain Kubernetes ClusterIP Service, each client opens one connection, lands on one pod, and sends it everything for hours; pods added by the autoscaler get nothing. The fixes all move balancing to the request level: an L7 proxy or mesh sidecar (Envoy, Linkerd, Istio) that balances individual streams, client-side balancing over a headless Service, or at least MAX_CONNECTION_AGE on the server so clients reconnect and re-spread periodically. WebSockets have the same shape and usually settle for least-connections plus periodic reconnects.

Session affinity pins a client to one backend, by a balancer-issued cookie (ALB AWSALB, HAProxy cookie) or by hashing the client IP. It exists to rescue apps that keep session state in process memory. The price is uneven load, a lost session whenever that backend dies or is deployed, and IP hashing that sends a whole corporate NAT to one instance. Move the state out, into a shared store or a signed token such as a JWT (JSON Web Token), and keep affinity only as a cache-locality optimisation, never as correctness.