Skip to content

Consumer apps at scale · 2 of 13

Step 1 · The pattern

A quiet day

People sign up at their own pace: 2 a second. The sign-up service works on 6 at a time, each in under a second, so it is never crowded. (All numbers are illustrative; one dot is a large group of people.)

  • visitor
  • signed up

In short

Virtual waiting room: how it works and how it breaks

A TV advert, a product drop or a celebrity post can send millions of people to the same page within a minute. Autoscaling takes minutes to add servers; the surge is over by then. If every visitor is let in, the service is overloaded: each request takes longer, many fail, and people press refresh, which adds more load. Overloaded services complete fewer requests than healthy ones, so letting everyone in serves fewer people. A virtual waiting room admits only as many visitors as the service can handle and puts the rest in a queue page served cheaply from the edge, letting them in as places free up. Everyone who is admitted finishes quickly, the service completes more per second, and the people waiting see their place in line instead of an error.

What this simulation simplifies

  • The service has two speeds: normal, and slow once more than ten are inside. A real service degrades gradually as threads, connections and the database contend.
  • The waiting room admits whenever a place is free. Real ones admit by configured rates, measured over seconds to minutes.
  • One dot stands for a large group of people, and every error page is refreshed within about a second.

How this comes up in interviews

"Marketing has bought a TV advert that will send millions of people to sign up in the same minute. How do you keep the sign-up flow working?"

Then they ask

  1. Why not just autoscale for the advert?
  2. How do you choose how many people to let in?
  3. What does someone in the queue see, and what happens when they refresh?

A strong answer covers

  • Autoscaling reacts in minutes; the surge lasts one, so capacity must be planned and admission controlled.
  • An overloaded service completes fewer requests than a healthy one, and refreshes add load.
  • A waiting room at the edge admits up to the service's tested capacity and queues the rest on a cheap page.
  • Set the active-users and new-users-per-minute limits from a load test, and make the queue token survive a refresh.
  • Serve the landing page statically from the edge, and pre-scale the parts you can.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.