Step 1 · The pattern
Every answer holds GPU memory
While the model writes an answer, the server keeps a cache entry for every token of that request on the GPU, so no token is worked out twice. Short requests hold a little memory; a long one holds several times more. All of it is freed when the answer is done. The numbers are illustrative.
- short request
- long request
- answered
In short
KV cache memory limits: how it works and how it breaks
A language model writes one token at a time, and each new token depends on everything before it. To avoid redoing that work, the server keeps a key-value (KV) cache: stored results for every token the request has seen so far. The cache lives in GPU memory, next to the model weights, and it grows as the prompt gets longer and as the answer is written. So memory, not compute, often decides how many requests a GPU can run at once, and one long request can cost as much as many short ones. A limit of eight requests at a time says nothing about memory. Admit by tokens instead: add up the prompt and output limit of every request in flight, and make new requests wait when that budget is spent. Waiting costs a little time. Running out of memory can lose every answer in progress.
What this simulation simplifies
- Running out of memory is drawn as the model process failing. A server that sizes its cache at start-up, as vLLM does, pauses requests instead; the crash is what happens without that protection.
- The gigabytes and the 22k token budget are made up. Real cache size per token depends on the model: work it out as described under Go deeper.
- Shared prefixes are not modelled. Every request is charged for its whole prompt, which overstates memory when many requests begin the same way.
How this comes up in interviews
"Your model server allows a fixed number of concurrent requests, yet it runs out of GPU memory when one customer sends long documents. Why, and what do you change?"
Then they ask
- Output length is unknown in advance. Do you reserve memory for the maximum or let the cache grow on demand?
- Memory runs short in the middle of generation. What should the server do instead of failing?
- Many requests start with the same system prompt. How does that change your token budget?
A strong answer covers
- The KV cache holds stored results for every prompt and output token, in GPU memory next to the weights.
- Memory per request follows its token count, so a request count is the wrong unit for admission.
- Admit by tokens in flight: each prompt plus its output limit, against a budget sized from the memory left after the weights.
- Every request needs an output limit, or the budget has to assume the whole context window.
- Waiting costs a little time. Running out of memory can lose every answer in progress.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Kwon et al.: Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) ↗ · vLLM documentation ↗ · vLLM docs source: Optimization and tuning (preemption) ↗ · vLLM docs source: Automatic prefix caching (design) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.