Skip to content

Rate limiter · 4 of 6

Step 1 · The pattern

One limit, three servers

The promise is 10 requests per window for this client. The API runs on three servers, and the load balancer deals requests out in turn, so each server sees only a third of them. At this pace nothing is refused.

  • request
  • served

In short

Distributed rate limiting: how it works and how it breaks

A rate limit is a promise about the whole system: this client gets 10 requests per window. But an API runs on many servers behind a load balancer, and each sees only its share of the traffic. If every server keeps its own count, each one allows the full limit, and the client gets the limit once per server. The count has to live in one place that all servers can reach, usually Redis. That alone is not enough. If a server reads the count, compares it, then writes it back, several requests arriving together all read the same old number and all pass. The check and the increment must be one atomic step: INCR adds one and returns the new count, and the server allows the request only if that number is within the limit. The key's expiry is set in the same step, so the window always ends.

How this comes up in interviews

"Your API runs on many servers behind a load balancer. How do you enforce one rate limit per client across all of them?"

Then they ask

  1. Each server reads the shared count, compares it, then writes it back. What goes wrong when requests arrive together?
  2. The shared store is unreachable. Does the limiter let requests through or refuse them?
  3. A call to the store on every request adds latency. How would you reduce that cost, and what accuracy do you give up?

A strong answer covers

  • Each server sees only its share of the traffic, so separate counts give the client the limit once per server.
  • Keep the count in one place that every server can reach, usually Redis.
  • Check and increment must be one atomic step: INCR returns the new count, and the request passes only if that is within the limit.
  • Set the key's expiry in the same step, with a transaction or a script, so the window always ends.
  • Decide in advance, per endpoint, whether the limiter fails open or closed when the store is down.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.