Step 1 · The pattern
A proxy in front of the model
A prompt goes through a proxy to the model, and the answer streams back piece by piece. Like most web infrastructure, the proxy has a time limit per request. Short answers never notice it.
- prompt / token in flight
- delivered
In short
Long-generation timeouts: how it works and how it breaks
Web infrastructure assumes a request is over in seconds, so proxies, load balancers and HTTP clients carry limits of 30 or 60 seconds on the whole request. A long model answer can legitimately run for minutes. A total limit then cuts a stream that was going fine. The client sends the prompt again, the model starts again from the first token, and the limit cuts it at the same point: the request can never succeed. Meanwhile the model is still writing the answers nobody is connected to receive. For streamed answers, limit silence instead of duration: an idle timeout that every token resets. When a caller disconnects, cancel the generation upstream. Cap the output length on purpose. And when the work is too long to hold a connection open for, make it a job: accept it, return an id, and let the client poll for the result.
What this simulation simplifies
- The clock is compressed: the 30-second limit and the 90-second answer play out in a few seconds of animation.
- One proxy stands in for every hop. A real path has several, each with its own timeouts, and the shortest one decides.
- Cancellation is instant. In practice a disconnect has to be noticed and passed upstream, and some serving stacks do not do that.
How this comes up in interviews
"A streamed model answer can run for minutes, but the proxy in front has a 60 second limit on the whole request. What happens, and how should timeouts be set?"
Then they ask
- The model is silent for a long time before its first token. What does an idle timeout do then, and how do you avoid it?
- The caller closes the tab mid-answer. What should happen upstream, and how do you check that it does?
- Some work takes longer than any connection will reliably stay open. How do you change the API?
A strong answer covers
- A total timeout cuts a stream that was going fine. The retry starts from the first token and is cut at the same point.
- For streamed answers, limit silence: an idle timeout that every token resets.
- Cancel the generation upstream when the caller disconnects, or the model keeps writing for nobody.
- Cap the output length on purpose.
- When the work is too long to hold a connection open, make it a job: accept it, return an id, let the client poll.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: nginx: proxy_read_timeout (ngx_http_proxy_module) ↗ · Claude API docs: Errors, section “Long requests” ↗ · Claude API docs: Streaming messages (ping events) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.