Step 1 · The pattern
Ask, wait, get the answer
Each user has one connection through a proxy to the model. The proxy closes a connection that carries nothing for 3 seconds. These answers are short, so each arrives whole before that matters.
- question
- open, silent
- answer
In short
Token streaming: how it works and how it breaks
A language model writes its answer one token at a time, so a long answer takes many seconds. If the server waits and sends the finished answer in one response, three things go wrong. The user looks at an empty screen for the whole generation. The connection carries no bytes, and proxies and load balancers on the path close connections that stay silent for too long. The user then presses the button again, and the model starts the same expensive answer a second time. Streaming sends each small piece as soon as it exists, usually as server-sent events on one open HTTP response. The first words show up almost at once, the connection is never silent for long, and a user who has read enough can stop, which lets the server stop generating. Total generation time is unchanged. Timings in the animation are illustrative.
What this simulation simplifies
- The proxy's idle limit is three seconds so that it fits the animation. Real idle limits are much longer, and so are real long answers.
- Each chip stands for a chunk of several tokens, sent every half second. Real token rates vary with the model and with load.
- Pressing stop always ends generation here. On a real stack you have to check that a disconnect is passed upstream: see Go deeper.
How this comes up in interviews
"A long model answer takes many seconds to write. How do you deliver it to a browser, and what breaks if you send it as one response?"
Then they ask
- Streaming is on and the server logs show tokens leaving steadily, yet users get the whole answer at once. Where do you look?
- The model fails halfway through an answer, after the success status was sent. How does the client find out?
- A stream drops mid-answer. What should the client do, and when is an automatic retry unsafe?
A strong answer covers
- Send each piece as soon as it exists, usually as server-sent events on one open HTTP response.
- The first words arrive almost at once and the connection is never silent for long. Total generation time is unchanged.
- Every hop must pass bytes on as they arrive: one buffering proxy or CDN turns the stream back into one late answer.
- A user who has read enough can stop, and the server should then stop generating.
- Measure time to first token at the client, not at the server.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: HTML Standard: Server-sent events (WHATWG) ↗ · Streaming messages (Claude API docs, Anthropic) ↗ · Streaming API responses (OpenAI API docs) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.