Skip to content

LLM inference serving · 1 of 6

Step 1 · The pattern

One request, one pass

Prompts wait in a queue and the GPU answers them one at a time. When it is quiet this looks fine: nobody waits. But look at the bar under the GPU. One request uses a small part of what the GPU could do in the same pass. (All numbers here are illustrative.)

  • request
  • answered

In short

Batching vs latency: how it works and how it breaks

A GPU can work on several requests in one pass in nearly the time it needs for one. A server that runs one request at a time wastes most of the GPU: few requests are served each second, and under moderate load a queue builds and waiting time climbs. Batching fixes the throughput, but the simple form has a cost. A static batcher holds the first arrival until enough requests have gathered, and with text generation the whole batch is busy until its longest answer is finished. Servers built for language models batch continuously instead. The model produces text one step at a time, so a waiting request can join the running batch at an upcoming step, and a finished one leaves and frees its place. A small limit on how long a request may wait keeps the delay bounded. The numbers in the animation are illustrative.

What this simulation simplifies

  • Every request costs the same here. Real requests differ in prompt and answer length, and a new request's prompt has to be processed before it can join the batch.
  • The GPU is drawn with eight places. The real limit is memory, counted in tokens, and throughput flattens before that limit is reached.
  • Arrival rates, timings and queue sizes are chosen so the animation is readable. None of them was measured on hardware.

How this comes up in interviews

"You serve a language model on one GPU and requests queue under moderate load. How do you raise throughput without ruining time to first token?"

Then they ask

  1. Traffic is light at night and a static batch takes seconds to fill. What do you change?
  2. A new request needs its whole prompt processed before it can join. What does that do to the streams already running?
  3. Throughput has stopped rising as you raise the batch size. What is happening, and what do you measure?

A strong answer covers

  • One request per pass wastes the GPU: a pass over several requests takes nearly the time of one.
  • A static batch holds the first arrival until it fills, and holds finished answers until the longest one ends.
  • Continuous batching lets a request join at an upcoming step and leave when it is done.
  • Cap how long a request may wait, and bound the queue so waits never outlast the caller's timeout.
  • Track time to first token and token rate separately: admitting eagerly helps one and hurts the other.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.