Skip to content

LLM inference serving · 3 of 6

Step 1 · The pattern

Few places, two kinds of work

A GPU has only a few places: here 4 requests at a time. Chat users send short requests that take half a second. A batch job sends long ones that hold a place for 4 seconds. A new GPU needs minutes to load the model, so you cannot add one when a spike arrives. (All numbers are illustrative.)

  • request
  • answered

In short

GPU queueing: how it works and how it breaks

A GPU can work on only so many requests at once, and you cannot add one quickly: a new replica has to load many gigabytes of model weights before it serves anything. So requests queue. A single first-come-first-served queue with no limit fails in three ways. A few long generations hold every place while many short ones wait behind them. Under overload the line grows until everyone waits longer than their caller will. And the server keeps generating for callers who have already given up, which wastes the capacity it is short of. Three changes fix this. Give the queue a limit and answer "busy, retry later" at once when it is full. Separate short interactive work from long batch work, with places kept for the interactive kind. And cancel a request the moment its caller disconnects. Batch work finishes later; chat keeps working.

What this simulation simplifies

  • One GPU with four places stands in for a fleet of replicas behind a load balancer.
  • A request holds its place for a fixed time. On a real server requests share a batch, and how fast each one runs depends on what else is running.
  • Every caller gives up after the same 2.4 seconds. Real deadlines differ by client, and some callers retry.

How this comes up in interviews

"During a spike your GPUs show 100% use, yet almost no chat request is answered in time. What is going on, and how do you redesign the queue?"

Then they ask

  1. A batch job submits many long generations at once. How do you keep chat responsive?
  2. Callers sit behind a gateway that keeps the connection open after they give up. How do you stop the wasted work?
  3. A new replica needs minutes to load its weights. What does that mean for autoscaling and spare capacity?

A strong answer covers

  • With an unlimited queue, waits grow past callers' patience and the GPU generates answers nobody will read.
  • Cap the queue so the longest wait is shorter than the caller's deadline, and refuse the rest at once with busy, retry later.
  • Separate short interactive work from long batch work, and keep places for the interactive kind.
  • Cancel a request when its caller disconnects or its deadline passes, in the queue and during generation.
  • Capacity cannot be added in seconds, so the queue and warm spare capacity must cover one cold start.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.