Skip to content

Load balancing & consistent hashing · 1 of 6

Step 1 · The pattern

Take the servers in turn

Three identical servers sit behind a load balancer. Round robin sends the first request to server 1, the next to server 2, the next to server 3, and starts again. Each gets a third of the requests, and the load bars match.

  • request
  • served

In short

Round robin vs least connections: how it works and how it breaks

A load balancer has to pick a server for every request. Round robin is the simplest rule: take the servers in turn, so each gets the same number of requests. That is fair only when every request costs about the same and every server is equally fast. Real fleets break both rules. Some requests are heavy, and one machine may be older or busy with other work. Round robin does not look. It keeps handing that machine its full share, so work piles up there and times out while the others have room to spare. Least connections uses one extra fact the balancer already has: how many requests it has sent to each server that have not been answered yet. Each new request goes to the server with the fewest. A slow server holds on to its requests for longer, so its count stays high and it is given fewer.

How this comes up in interviews

"You have a pool of servers behind a load balancer and one of them is slower than the rest. How should the balancer choose a server for each request?"

Then they ask

  1. One server starts returning errors within a few milliseconds. What does least connections do with it, and how do you stop that?
  2. A new server joins the pool with zero connections. What happens to it, and what would you change?
  3. You run many balancer instances and requests finish in milliseconds. Is the connection count still a useful signal?

A strong answer covers

  • Round robin gives every server the same number of requests, which is fair only if requests cost about the same and servers are equally fast.
  • A slow server still gets its full share under round robin, so its queue fills and requests time out while the others have room.
  • Least connections picks the server with the fewest unanswered requests. A slow server holds requests longer, so it is given fewer.
  • The count says nothing about whether answers are good: a fast-failing server looks idle, so pair it with health checks and error tracking.
  • Refinements: weights for known hardware differences, slow start for a fresh server, and picking the less busy of two random servers.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.