Step 1 · The pattern
Room to spare
A checkout API has six workers. Each request takes one worker for about half a second. At normal load some workers are always free, so every request starts at once and comes back green. Error rate: 0%.
- request
- served
In short
Saturation vs errors: how it works and how it breaks
An error tells you something has already failed. Saturation tells you how full a resource is and how much work is waiting for it. The two move at different times. A service pushed past its limit does not fail straight away. Its workers are all busy, so new requests wait. The queue grows, every request gets slower, and the error rate is still zero. Only when the queue is full, or requests have waited longer than their timeout, do errors appear, and then they appear all at once. A team that alerts only on the error rate hears nothing until users are already failing. A team that also watches saturation, meaning the queue length or the time spent waiting, sees the same problem minutes earlier, while there is still time to add capacity or shed load. Errors confirm the damage. Saturation predicts it.
How this comes up in interviews
"Traffic to a service climbs slowly past what its workers can serve. Walk me through what each dashboard signal does, and what you would alert on."
Then they ask
- A worker pool is 100% busy. Is that a problem? How do you tell busy from overloaded?
- Would you page a person on queue length? If not, what do you page on, and where does saturation go?
- Queue length or wait time: which is the better signal, and how do you choose the threshold?
A strong answer covers
- Past its limit a service does not fail straight away: workers are busy, requests wait, and the error rate is still zero.
- Errors appear all at once, when the queue is full or requests have waited longer than their timeout.
- Saturation is the work waiting for a resource: queue length, or time spent waiting.
- It moves minutes before errors do, while there is still time to add capacity or shed load.
- An average of utilisation smooths over the moments when the pool was full.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Brendan Gregg: The USE Method ↗ · Google SRE book: Monitoring Distributed Systems (the four golden signals) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.