Skip to content

Resilience patterns · 1 of 9

Step 1 · The pattern

Half busy, short queue

One server handles one request at a time: most take 200 ms, one in five takes 1.5 s (a cache miss, a big page), so it can serve about 2.2 a second. Requests arrive at random, 1.1 a second: the server is busy about 50% of the time. Sometimes two or three arrive together and wait briefly, and the idle moments clear them. (All numbers are illustrative.)

  • request
  • served

In short

Queueing and utilisation: how it works and how it breaks

A server that is busy half the time has a short queue: requests arrive at random, and the idle half absorbs the bursts. As the arrival rate approaches what the server can handle, the idle time that cleared those bursts disappears, and waits grow sharply rather than in proportion. In this animation, going from 50% to 80% busy more than doubles the queue, and 95% doubles it again, with bursts that reach the limit and are refused. Little's law ties the numbers together: the average number of requests in the system equals the arrival rate times the average time each spends there, so a longer wait means a longer queue at the same traffic. Plan capacity for a target utilisation well below 100%, watch queue length and latency rather than only CPU, and shed load before the queue runs away.

What this simulation simplifies

  • One server handles one request at a time, and requests take either 200 ms or 1.5 s; real services have many workers and a wider spread of request times.
  • A request that finds twelve already in the system is refused immediately; real clients wait until their own timeout.
  • Doubling capacity is shown as a second identical server; adding workers, faster code or shedding low-priority work moves the same curve.

How this comes up in interviews

"Your service runs at 75% CPU on average and the team wants to save money by running at 95%. What do you tell them?"

Then they ask

  1. Why does latency rise so sharply near full utilisation?
  2. How would you use Little's law to sanity-check a capacity plan?
  3. What would you monitor to see the knee coming?

A strong answer covers

  • Waits grow non-linearly with utilisation; near 100% the idle time that absorbs bursts disappears.
  • Little's law: in-flight requests = arrival rate × time in the system, so latency and concurrency move together.
  • Pick a target utilisation that keeps tail latency within the SLO at peak, including headroom for bursts and failures.
  • Monitor queue depth, wait time and p99, not just average CPU.
  • Shed or reject excess work early rather than letting queues grow.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Engineering write-up

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.