Step 1 · The pattern
Three kinds of request
Devices send a mix of requests through one gateway. A few are critical: without them the user cannot press play. Some are extras, such as viewing history. The rest is background noise like log uploads. On a quiet day all of them are served.
- critical (play)
- extras
- background
- served
In short
Prioritised load shedding: how it works and how it breaks
Every server has a limit. Pushed past it, work queues up, each request takes longer, and soon requests fail at random. The server cannot tell a request the product depends on from a background log upload, so both fail at the same rate. Load shedding means refusing some work on purpose so that the rest succeeds. Prioritised load shedding decides which work to refuse. A gateway in front of the backend ranks every request by what breaks for the user if it fails. When the backend shows trouble, such as rising errors or too many requests in flight, the gateway turns away the lowest priority first and moves up the ranking only if the trouble grows. Shed requests get a quick answer telling the client to back off, which also stops a retry storm. Critical requests are usually a small share of traffic, so they keep working.
How this comes up in interviews
"A bad client release floods your backend with background requests, and users can no longer press play. How do you keep the critical path working?"
Then they ask
- How do you decide the ranking, and who is allowed to set a request's priority?
- Which signal tells the gateway to start shedding, and how does it decide how far up the ranking to go?
- What do shed requests receive, and how do you stop the refusals turning into a retry storm?
A strong answer covers
- Past its limit a server queues work and fails requests at random. It cannot tell a critical request from a background log upload.
- Load shedding refuses some work on purpose so that the rest succeeds. Prioritised load shedding decides which work to refuse.
- A gateway ranks every request by what breaks for the user if it fails, and turns away the lowest priority first.
- Shed requests get a quick answer telling the client to back off, which also stops a retry storm.
- Critical requests are usually a small share of traffic, so they can be kept. Random shedding rejects them at the same rate as the rest.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Netflix Tech Blog: Keeping Netflix reliable using prioritized load shedding ↗ · RFC 9110: HTTP semantics (503 and Retry-After) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.