Use case · Streaming
Live event kick-off
Everyone presses play in the same minute. Watch one viewer start a live stream, a crowd flood the origin through cold caches, then the fix: preparation and back-off.
Used by apps like Hotstar, Netflix, Prime Video, YouTube Live and DAZN
Happy path · break it · fix it
Live event kick-off: sequence diagram
Live event kick-off, shown as messages between Viewer (The video player in a viewer's app or TV), Auth (The login and entitlement service, which decides whether this account may watch the event), Manifest (The manifest service, which hands out the playlist: the list of the newest video segments), CDN edge (The CDN edge servers near viewers, which cache video segments), Origin (The origin servers, which package the live video into segments and serve them to the CDN).
Happy path: 9 steps.
- Viewer → Auth: play?. A viewer presses play a few minutes early, and the app asks whether this account may watch.
- Auth → Viewer: token (reply). The entitlement check passes and returns a short-lived playback token.
- Viewer → Manifest: GET playlist. With the token, the player asks for the playlist of the live stream.
- Manifest → Viewer: segments 41–43 (reply). The playlist names the newest few seconds of video; the player will ask again shortly.
- Viewer → CDN edge: GET segment 41. The player fetches the first segment from the CDN edge nearest to it.
- CDN edge → Viewer: HIT · segment 41 (reply). Earlier viewers already pulled it through, so the edge answers from its cache.
- Viewer → Manifest: GET playlist. The timer fires: a live playlist keeps growing, so the player asks for it again.
- Manifest → Viewer: segments 42–44 (reply). One new segment has appeared, and playback carries on without a gap.
- CDN edge: 1 origin fetch each. Each new segment is fetched from the origin once, then served to everyone nearby from the edge.
Result: Video plays, and the origin sees one request per segment per edge.
Break it: 10 steps.
- Viewer → Auth: play? (many at once). Kick-off: everyone presses play in the same minute.
- Auth: checks pile up. Every play needs an entitlement lookup, so the checks queue and answers slow down.
- Auth → Viewer: token (reply) (many at once). The tokens arrive late, and all at about the same moment.
- Viewer → Manifest: GET playlist (many at once). The whole crowd asks for the playlist together.
- Manifest → Viewer: segments 1–3 (reply) (many at once). Everyone is pointed at the same brand-new segments.
- Viewer → CDN edge: GET segment 1 (many at once). Every player asks its edge for a segment that is only seconds old.
- CDN edge → Origin: MISS (many at once). No edge has it yet, so every request falls through to the origin at the same time.
- Origin → CDN edge: timeout (never arrives) (reply) (many at once). The origin has more fetches than it can serve, and most of them time out.
- Viewer → CDN edge: retry now (many at once). Players retry at once, each retry is one more origin fetch, and autoscaling starts new servers.
- Origin: ready, but too late. The new servers take too long to become ready; by then the retries have multiplied the load.
Result: Viewers watch a spinner while the origin drowns in retries.
Fix it: 10 steps.
- an hour before kick-off. Capacity is raised ahead of time to the peak seen at past events, and the edges are warmed up.
- Viewer → Auth: play? (many at once). The same kick-off, the same crowd, the same minute.
- Auth → Viewer: cached OK (reply) (many at once). Entitlement was checked earlier and cached, so the answer comes back without a lookup.
- Viewer → Manifest: GET playlist (many at once). The crowd asks for the playlist together, as before.
- Manifest → Viewer: list + back off (reply) (many at once). Panic mode: the reply tells players to skip extras such as recommendations and to back off.
- Viewer: random waits: 1s, 6s. Each player picks its own random wait, so the crowd no longer moves in step.
- Viewer → CDN edge: GET segment 1. The first player's short wait ends, and it asks its edge for the segment.
- CDN edge → Origin: 1 fetch. The edge fetches each new segment once, however many players are asking for it.
- Origin → CDN edge: segment 1 (reply). The origin answers quickly, with capacity to spare.
- CDN edge → Viewer: HIT · segment 1 (reply) (many at once). Players that waited longer are served from the edge cache; the origin never sees them.
Result: The crowd is spread over seconds and the origin stays inside capacity.
step 1 of 9
A viewer presses play a few minutes early, and the app asks whether this account may watch.
- request
- success
- held / reserved
- waiting / retrying
At scale
One viewer never shows this failure. Turn up the crowd and compare a cold edge with instant retries against a warm edge with back-off. All numbers are illustrative.
Numbers are illustrative: a small simulation, not a measurement.
The happy path, step by step
- A viewer presses play and the app asks the entitlement service whether this account may watch the event.
- The check passes and the app receives a short-lived playback token.
- The player asks the manifest service for the playlist of the live stream.
- The playlist lists the newest few segments of video, each a few seconds long.
- The player fetches the first segment from the nearest CDN edge.
- The edge already holds that segment, since an earlier viewer pulled it from the origin, and answers from its cache.
- A few seconds later the player asks for the playlist again, since a live playlist keeps growing.
- The new playlist names the next segment, the player fetches it, and playback continues.
- Each new segment travels from the origin to an edge once and is then served to every viewer near that edge.
The failure
What breaks and why
A film can be copied to every edge days ahead; a live segment does not exist until seconds before it is watched. At kick-off millions of players press play in the same minute: entitlement checks queue, and then every player asks for the same brand-new segments, which no edge holds yet, so the requests fall through to the origin together. The origin saturates and fetches time out. Players that retry immediately add a second and third wave on top of the first, and servers started by reactive autoscaling become ready only after the damage is done. Viewers see a spinner at the one moment they care about.
Before / after
The fix
Prepare for the step change, then shape the crowd. Before the event, scale every tier to the peak forecast from past events, so nothing depends on new servers arriving in time. Warm the edges and let each edge fetch a new segment once, however many players ask. Cache entitlement results so play does not mean a lookup. Give the server a panic switch: under load, replies tell players to wait a random, growing time before asking again and to skip extras such as recommendations, which leaves the capacity for video.
onPlay():
loadRecommendations() // extras first
seg = fetch(nextSegment)
while seg.failed:
seg = fetch(nextSegment) // again, now
show(seg)
// a million players do this in steponPlay():
r = fetch(playlist)
if not r.panic: loadRecommendations()
wait = r.backoffMs or 500
seg = fetch(nextSegment)
while seg.failed and tries < 5:
sleep(random(0, wait)) // jitter
wait = min(wait * 2, 30_000)
seg = fetch(nextSegment); tries += 1
show(seg)
// same players, spread over secondsBoss fight
Diagnose the real outage
The real outage
Based on reporting, no official postmortemA record crowd tuned in for a live boxing match and many saw error screens. Films on the same service played fine, and the internet providers said their links had room. Where do you look?
Fight the boss90 seconds · the company is revealed after you answer
From their own write-ups
How real companies approach it
Hotstar writes that auto-scaling is too slow for the spikes of live cricket, so it estimates peak concurrency and scales up before the event. It keeps auto-scaling only to hold the right number of servers in a pool, and adds more ahead of need when headroom thresholds are crossed.
Hotstar: Scaling Is Not An Accident (2018) ↗Hotstar describes “panic” switches that tell its client apps to ease off for a moment, with exponential or custom back-off, so that jitter gives the back end time to heal. It also says its clients add jitter to requests, and that single features can degrade while video keeps playing.
Hotstar: Scaling Is Not An Accident (2018) ↗Hotstar says that for a 50 million target it agreed an order in which services are turned off as concurrency climbs, so that viewers can still get in and video keeps playing. It also warns that many cached objects expiring together can send a burst of requests to the origin.
Hotstar: Scaling the Hotstar Platform for 50M (2019) ↗
How this comes up in interviews
"Millions of viewers press play in the same minute at kick-off. Design the start of a live stream so that it holds."
Then they ask
- Why can you not rely on autoscaling for this spike?
- No edge holds the newest segment yet. How do you stop every player's request reaching the origin?
- The origin is overloaded anyway. What should the server tell the players to do?
A strong answer covers
- A live segment does not exist until seconds before it is watched, so it cannot be copied to the edges ahead of time.
- Scale every tier to the forecast peak before the event: servers started by reactive autoscaling are ready after the damage.
- Warm the edges and let each edge fetch a new segment once, however many players ask.
- Cache entitlement results so that pressing play does not mean a lookup.
- A panic switch tells players to wait a random, growing time and to skip extras, which leaves the capacity for video.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
Your turn
What would you do?
A cup final starts at 20:00. At 20:00:30 origin CPU is at 100%, segment fetches time out, players retry instantly, and the autoscaler's new servers are still starting. Which plan would have kept the stream up?
Related
Live event kick-off questions
- Why do live streams buffer at the start of a big event?
- Millions of players start in the same minute and all ask for video segments that were created seconds ago, so no CDN edge has them cached yet. The requests reach the origin together, it slows down or times out, and players that retry at once make the peak higher.
- Why does autoscaling not handle a live event spike?
- Autoscaling reacts after load has risen, and a new server needs time to start and pass its health checks. At kick-off the load multiplies within seconds, so the extra servers arrive after the overload. Streaming teams therefore scale up before the event, from a forecast of the peak.
- What is panic mode in a streaming app?
- It is a switch on the server that tells client apps the back end is under heavy load. Clients then wait longer and at random before asking again, and drop non-essential calls such as recommendations, so that the remaining capacity goes to playing video.
Sources: Hotstar: Scaling Is Not An Accident (2018) ↗ · Hotstar: Scaling the Hotstar Platform for 50M (2019) ↗ · StreamingMediaBlog (Dan Rayburn): Some tech details from Netflix's boxing stream (industry reporting, Dec 2024) ↗ · StreamingMediaBlog (Dan Rayburn): Netflix details its live streaming infrastructure (Jul 2025) ↗ · RFC 8216: HTTP Live Streaming (playlists, media segments, reloading a live playlist) ↗ · Exponential Backoff And Jitter (AWS Architecture Blog, 2015) ↗ · Report a problem
A typical design, informed by public write-ups, not any one company's exact system. Diagrams are our own. Numbers in the simulation are illustrative. All wording is original and paraphrases the linked sources.