Step 1 · The pattern
A fleet sized for the plan
Each server handles 2 requests at a time. Two servers serve the planned 3 requests a second with room to spare. The autoscaler checks once a second and would start a server if the fleet were nearly full. (All numbers are illustrative, and time is compressed.)
- request
- served
In short
Autoscaling lag: how it works and how it breaks
Autoscaling adds servers after it measures load, and each new server needs time to start: boot, pull the image, load configuration and warm caches. On a normal day traffic grows slowly enough. On launch day it can arrive at many times the plan within minutes. One game launch described targeting a worst case of five times its expected traffic and getting fifty. Requests beyond capacity fail, users retry, and the retries add load while the new servers are still starting, so the fleet is always behind. Plan capacity for the worst case you can justify and add a margin. Keep a warm pool of servers that are already started, so each scaling step takes seconds. Measure how long a cold start really takes, and shed or queue less important work while the fleet catches up.
What this simulation simplifies
- Time is compressed: a cold start takes 8 seconds here and a warm one 1.5. Real cold starts take minutes, warm ones seconds to a minute.
- The autoscaler starts one server at a time when the fleet is nearly full. Real policies can add several at once, based on a target such as CPU or requests per server.
- Every refused request is retried within about a second, and nothing else in the path (database, cache) limits the fleet.
How this comes up in interviews
"You are launching a consumer app with a big marketing push. Traffic could be far above the forecast. How do you prepare the infrastructure?"
Then they ask
- Autoscaling is enabled. Why is that not enough on its own?
- How do you decide how much capacity to have running before launch?
- Traffic arrives at ten times the worst case anyway. What do you do in the first ten minutes?
A strong answer covers
- Autoscaling reacts after load arrives, and each server takes time to start; a launch can outrun it.
- Retries during the gap add load and keep the fleet full.
- Start with headroom for a justified worst case, and keep a warm pool so each scaling step takes seconds.
- Measure real start time and scale every layer in the path, including the load balancer and the database.
- Plan what to shed, queue or switch off if traffic exceeds even the worst case.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Google Cloud blog: Bringing Pokémon GO to life on Google Cloud (2016) ↗ · Amazon EC2 Auto Scaling: Decrease latency for applications with long boot times using warm pools ↗ · Kubernetes documentation: Horizontal Pod Autoscaling ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.