Skip to content

Rate limiter · 3 of 6

Step 1 · The pattern

A queue with a hole in it

Checkouts go into a bucket in front of the payment provider. The bucket lets one out every 200 ms, never faster. With calm traffic nothing queues: each request waits its turn for a moment and goes on.

  • waiting in bucket
  • released
  • charged

In short

Leaky bucket: how it works and how it breaks

A leaky bucket is a small queue that drains at a fixed rate. Requests go in at whatever pace they arrive and come out one at a time, evenly spaced. A token bucket limits the average and lets a saved-up burst through in one go. That suits your own service, but not a dependency that fails when ten calls land together: a payment provider, an old database, a third-party API with its own limit. Put a leaky bucket in front of that dependency and it never sees a burst. Bursts wait in the bucket instead. The price is waiting time, so the bucket must have a fixed size. When it is full, new requests are refused at once with 429. The longest wait is the bucket size divided by the drain rate. A bucket with no size limit refuses nothing, but the wait grows until callers give up.

How this comes up in interviews

"Your checkout calls a payment provider that fails when many calls land together, although the average rate is fine. How do you protect it?"

Then they ask

  1. How big should the bucket be, and what is the longest a request can wait in it?
  2. What should happen to queued requests whose callers have already timed out?
  3. Where would you use a token bucket instead, and why not hold requests from users in a queue?

A strong answer covers

  • A leaky bucket is a small queue that drains at a fixed rate, so the dependency gets evenly spaced calls and never sees a burst.
  • A token bucket limits the average but passes a saved-up burst on in one go, which this dependency cannot take.
  • The price is waiting time, so the bucket has a fixed size and refuses new requests at once with 429 when it is full.
  • The longest wait is the bucket size divided by the drain rate, so choose the size from how long callers will wait.
  • Token bucket on requests from users, leaky bucket on the calls to the fragile dependency.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.