Step 1 · The pattern
A queue with a hole in it
Checkouts go into a bucket in front of the payment provider. The bucket lets one out every 200 ms, never faster. With calm traffic nothing queues: each request waits its turn for a moment and goes on.
- waiting in bucket
- released
- charged
In short
Leaky bucket: how it works and how it breaks
A leaky bucket is a small queue that drains at a fixed rate. Requests go in at whatever pace they arrive and come out one at a time, evenly spaced. A token bucket limits the average and lets a saved-up burst through in one go. That suits your own service, but not a dependency that fails when ten calls land together: a payment provider, an old database, a third-party API with its own limit. Put a leaky bucket in front of that dependency and it never sees a burst. Bursts wait in the bucket instead. The price is waiting time, so the bucket must have a fixed size. When it is full, new requests are refused at once with 429. The longest wait is the bucket size divided by the drain rate. A bucket with no size limit refuses nothing, but the wait grows until callers give up.
How this comes up in interviews
"Your checkout calls a payment provider that fails when many calls land together, although the average rate is fine. How do you protect it?"
Then they ask
- How big should the bucket be, and what is the longest a request can wait in it?
- What should happen to queued requests whose callers have already timed out?
- Where would you use a token bucket instead, and why not hold requests from users in a queue?
A strong answer covers
- A leaky bucket is a small queue that drains at a fixed rate, so the dependency gets evenly spaced calls and never sees a burst.
- A token bucket limits the average but passes a saved-up burst on in one go, which this dependency cannot take.
- The price is waiting time, so the bucket has a fixed size and refuses new requests at once with 429 when it is full.
- The longest wait is the bucket size divided by the drain rate, so choose the size from how long callers will wait.
- Token bucket on requests from users, leaky bucket on the calls to the fragile dependency.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: nginx: ngx_http_limit_req_module ↗ · Scaling your API with rate limiters (Stripe, 2017) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.