Step 1 · The pattern
A balancer that spreads connections
Three services call a pool of servers through a layer-4 balancer. It never reads a request. When a connection opens it picks a server, then copies bytes. Here every call opens its own short connection, so each call gets its own pick.
- call on a new connection
- served
In short
L4 vs L7 load balancing: how it works and how it breaks
A load balancer can work at two levels. A layer-4 balancer works with TCP connections: when a connection opens it picks a server, and from then on it only copies bytes. A layer-7 balancer reads the protocol inside, so it sees each request and can pick a server for every one. When each call opens its own short connection, the two give the same spread. HTTP/2 and gRPC change that. A caller opens one connection, keeps it for hours and sends all its requests down it at once. Behind a layer-4 balancer every one of those requests lands on the same server. A busy caller overloads its server while others idle, and a server you add gets nothing, because no existing connection leads to it. Balancing per request fixes this, either in a layer-7 proxy or inside the client.
How this comes up in interviews
"You scale a gRPC service from three pods to six. An hour later the old three are still hot and the new three are idle. Why, and what do you change?"
Then they ask
- What does moving to a layer-7 balancer cost you compared with copying bytes at layer 4?
- You must stay on layer 4. What partial fix is there, and which case does it not help?
- How does balancing inside the client compare with a proxy, and where would you still choose layer 4?
A strong answer covers
- A layer-4 balancer picks a server once, when a connection opens, then only copies bytes. A layer-7 balancer reads the protocol and picks per request.
- HTTP/2 and gRPC keep one connection open and send many requests down it, so behind layer 4 all of a caller's requests reach one server.
- A busy caller overloads its server while others idle, and a new server gets nothing because no existing connection leads to it.
- Fix it by balancing per request: a layer-7 proxy, or a client that holds a connection to every server and picks one per call.
- Layer 7 must end TLS and parse requests, so it costs more per request. A maximum connection age is a partial fix on layer 4.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: gRPC Load Balancing on Kubernetes without Tears (Kubernetes blog, 2018) ↗ · gRPC Load Balancing (gRPC blog, 2017) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.