Step 1 · The pattern
One door to the model
Chat and search belong to different teams, but every model call goes through one gateway. It holds the provider key, counts each tenant's tokens, and passes the call on. The provider limits the whole account by tokens per minute (a minute is 4 seconds here).
- model call
- answered
In short
LLM gateway: how it works and how it breaks
An LLM gateway is one service that sits between every team in your product and the model providers. Providers limit an account mostly by tokens per minute, not only by requests, so a single job with long prompts can spend the quota that everyone shares. Without a gateway, the chat feature then gets 429s for something another team did, and an incident at the provider stops every feature at once. The gateway holds the provider keys and gives each tenant its own token budget, so a heavy job waits on its own share. It notices provider errors and sends traffic to a second provider or model while the first is failing. It is also the natural place to count spend per tenant and to keep prompt prefixes stable, so the provider's prompt cache can be reused. The numbers in the animation are illustrative.
What this simulation simplifies
- A minute lasts four seconds, and the count restarts at each minute. Real provider limits refill continuously: see Go deeper.
- Every prompt costs a fixed number of tokens, and output tokens are not counted separately. Real budgets track input and output apart.
- The fallback provider answers exactly like the first. A real second model behaves differently and needs its own prompts and tests.
How this comes up in interviews
"Three teams call a model provider through one account. A nightly job starts and the chat feature gets 429s. Design what sits between the teams and the provider."
Then they ask
- Output tokens are unknown before the call. How does the gateway charge a tenant's budget?
- The provider has an incident. When do you send traffic elsewhere, and what can go wrong on the fallback model?
- The provider offers prompt caching. What can a gateway do that helps the hit rate, and what destroys it?
A strong answer covers
- Providers limit an account mostly by tokens per minute, so a few long prompts can spend the quota everyone shares.
- The gateway holds the provider keys and gives each tenant its own token budget: a heavy job waits on its own share.
- It notices provider errors and sends traffic to a second provider or model while the first is failing.
- It is the natural place to count spend per tenant and to keep prompt prefixes stable for the provider's cache.
- Retrying at once after a 429 adds load to an account that is already over its limit.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Claude API docs: Rate limits ↗ · Claude API docs: Prompt caching ↗ · LiteLLM docs: Router, load balancing and fallbacks ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.