Use case · Social & messaging
Chat reconnect storm
A scale-down cuts a share of chat sessions and every client reconnects in the same instant. Watch the herd take the gateways down, then spread it out.
Used by apps like Discord, Slack, WhatsApp, Microsoft Teams and Twitch chat
Happy path · break it · fix it
Chat reconnect storm: sequence diagram
Chat reconnect storm, shown as messages between Client (The chat app on a user's device), Gateway (The gateway servers that hold every client's WebSocket open), Sessions (The session service, which keeps one session per connected device), Messages (The database that stores message history).
Happy path: 10 steps.
- Client → Gateway: CONNECT wss. The app opens one WebSocket to a gateway and keeps it open.
- Gateway → Sessions: start session. The gateway asks the session service to start a session for this device.
- Sessions → Messages: load history. Starting from scratch is costly: the session loads channels and recent messages.
- Sessions → Client: READY seq=1 (reply). The client gets its starting state and a sequence number to count from.
- Client → Gateway: heartbeat. A small heartbeat every few seconds proves the connection is still alive.
- Sessions → Client: MSG seq=412. Each pushed message carries the next sequence number, so the client knows its place.
- Sessions: pod drains first. A planned scale-down hands each session to another server before the old one stops.
- clients move over 60 s. Gateways move their clients a few at a time, never all in one instant.
- Client → Gateway: RESUME 412. A moved client says the last sequence number it saw.
- Sessions → Client: resumed, 0 lost (reply). The session is still there, so only missed messages are sent: no full reload.
Result: Servers were removed and nobody noticed.
Break it: 10 steps.
- Client → Gateway: CONNECT wss (many at once). Many clients are connected, each with its own session.
- Sessions → Client: READY seq=1 (reply) (many at once). All of them are online and receiving messages.
- Sessions: pods killed, no drain. A scale-down stops session servers before they hand over, so their sessions vanish together.
- Gateway → Client: RECONNECT (many at once). The gateway tells every affected client to reconnect, all in the same instant.
- Client → Gateway: CONNECT wss (many at once). They all obey at once, and every one needs a costly new session.
- Sessions → Client: 429 limit hit (reply). The session-start limit refuses most of them, and the clients barely wait before trying again.
- Client → Gateway: CONNECT wss (many at once). The retries land together, on top of the clients still being set up.
- Gateway: out of memory. Preparing that many sessions at once uses up the gateway's memory, and it restarts.
- Gateway → Client: socket closed (many at once). The restart cuts off clients that were fine until now: a second, bigger herd.
- Client → Gateway: CONNECT wss (never arrives) (many at once). They reconnect at once too, into gateways that are still starting up.
Result: The trigger is long over. The reconnects keep the outage going.
Fix it: 10 steps.
- Client → Gateway: CONNECT wss (many at once). The same clients, connected as before.
- Sessions → Client: READY seq=1 (reply) (many at once). All of them are online and receiving messages.
- Sessions: pods killed, no drain. The same scale-down mistake: session servers stop before handing over.
- Gateway → Client: RECONNECT (many at once). Every client is told to reconnect, and each one first picks a random wait of its own.
- Gateway: typing, presence: off. While clients wait, the gateway switches off typing indicators and presence to save capacity.
- Client → Gateway: RESUME 412 (many at once). The first clients return with their last sequence number, not a request to start over.
- Gateway → Client: retry in 7s (reply). The gateway admits a fixed number per second and tells the rest exactly when to return.
- Sessions → Client: missed msgs only (reply). An admitted client gets only what it missed since 412, which is cheap to serve.
- Client → Gateway: RESUME 412 (many at once). Later clients arrive seconds apart, so the gateway always has room for them.
- Sessions → Client: all back, 0 lost (reply) (many at once). Everyone is back within a minute, and typing and presence are switched on again.
Result: Same mistake, a slow minute for some, no outage.
step 1 of 10
The app opens one WebSocket to a gateway and keeps it open.
- request
- success
- held / reserved
- waiting / retrying
At scale
Three clients reconnecting is nothing. Turn up the crowd, then watch what happens three seconds in, when a share of connections is cut. All numbers are illustrative.
Numbers are illustrative: a small simulation, not a measurement.
The happy path, step by step
- The app opens one WebSocket connection to a gateway server and keeps it open.
- The gateway asks the session service to start a session for that device.
- The session loads the channels and recent messages the client needs.
- The client receives its starting state and a sequence number.
- The client sends a small heartbeat every few seconds to show the connection is alive.
- New messages are pushed down the open connection, each with the next sequence number.
- Before a planned scale-down, a server hands its sessions to another server and only then stops.
- Gateways move their clients over a spread-out period, a few at a time.
- A moved client reconnects and resumes with the last sequence number it saw.
- The server sends only the messages the client missed.
The failure
What breaks and why
Every client holds a connection that is cheap to keep and costly to set up. When servers are removed without handing over first, a large share of sessions ends in the same instant and every one of those clients reconnects at once. That wave is far bigger than normal sign-in traffic, so it trips the session-start limit, and clients that retry immediately turn one wave into many. Gateways preparing thousands of sessions at once run out of memory and restart, which cuts off clients that were still fine and sends a second herd. After that the original mistake no longer matters: the reconnecting alone keeps the system down.
Before / after
The fix
Spread the crowd out, and make each return cheap. Clients wait a random delay before reconnecting and double the range after every failure, so arrivals trickle in. The gateway admits a fixed number of new sessions per second and tells everyone else when to come back, so it never takes on more than it can hold. A returning client resumes with its session id and last sequence number and gets only what it missed. Until the backlog clears, switch off typing indicators and presence. And drain a server before removing it.
socket.onclose = () => {
while (!connected) {
connect() // right away, every time
identify(token) // full re-sync from scratch
}
}socket.onclose = (info) => {
for (n = 0; !connected; n++) {
ceiling = min(CAP, BASE * 2 ** n)
sleep(random(0, ceiling)) // full jitter
sleep(info.retryAfter ?? 0) // server's hint
connect()
resume(sessionId, lastSeq) // only the gap
}
}Boss fight
Diagnose the real outage
The real outage
A routine resize ended one in six sessions at once. Text chat came back by itself. Voice stayed down for three hours, and a full restart of the stuck cluster bought only four minutes.
Fight the boss90 seconds · the company is revealed after you answer
From their own write-ups
How real companies approach it
Discord says that after its March 2026 voice outage it added a Kubernetes admission webhook that rejects a scale-down until the pod has handed off its processes, and that it tuned and added rate limits in front of its voice services.
Discord: Behind the scenes of the 3/25/26 voice outage ↗Discord describes turning off typing events and the marking of messages as read during its November 2023 authentication outage, to take load off an overloaded database cluster. It says this restored authentication for about half of its users.
Discord: The 11/6/23 authentication outage ↗Uber describes its push protocol giving every message a sequence number: a client that reconnects sends the largest number it has seen and the server continues from there. The server sends a heartbeat every 4 seconds, and a client reconnects after 7 seconds of silence.
Uber Engineering: Uber's real-time push platform ↗
How this comes up in interviews
"A scale-down drops a large share of your chat app's WebSocket connections at once. What happens next, and how do you design clients and gateways to survive it?"
Then they ask
- Clients back off with growing delays, but every client uses the same delays. Why is that not enough?
- A client returns after a long delay and its resume is refused. What happens then, and how does that shape your backoff?
- How should a server be removed in a planned scale-down so that no storm starts?
A strong answer covers
- A connection is cheap to keep and costly to set up, so a mass reconnect is far bigger than normal sign-in traffic.
- Clients wait a random delay before reconnecting and double the range after every failure.
- The gateway admits a fixed number of new sessions per second and tells everyone else when to come back.
- A returning client resumes with its session id and last sequence number and gets only what it missed.
- Switch off typing indicators and presence until the backlog clears, and drain a server before removing it.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
Your turn
What would you do?
A config change cuts 15% of your chat app's connections at once. Clients reconnect immediately, the session-start limit trips, gateways run out of memory and restart, and every restart cuts off more clients. Which change breaks the loop?
Related
Chat reconnect storm questions
- What is a reconnect storm?
- It is what happens when a large number of clients lose their connection at the same moment and all try to reconnect at once. Setting up a connection costs far more than keeping one open, so the wave can overload the servers, whose failures then disconnect even more clients.
- Why add jitter to reconnect backoff?
- Backoff alone makes every client wait the same time, so they all return together a little later. Jitter makes each client pick a random wait inside the backoff range. The same number of clients then arrives spread over seconds or minutes, at a rate the servers can take.
- What does resuming a session with a sequence number mean?
- The server numbers every event it pushes to a client. After a short disconnect the client sends its session id and the last number it received, and the server replays only what came after. That is much cheaper than starting a new session and loading everything again.
Sources: Discord: Behind the scenes of the 3/25/26 voice outage ↗ · Discord: The 11/6/23 authentication outage ↗ · Discord developer documentation: Gateway (heartbeats, sequence numbers, resuming) ↗ · Slack Engineering: Real-time messaging ↗ · Uber Engineering: Uber's real-time push platform ↗ · Exponential Backoff And Jitter (AWS Architecture Blog, 2015) ↗ · Report a problem
A typical design, informed by public write-ups, not any one company's exact system. Diagrams are our own. Numbers in the simulation are illustrative. All wording is original and paraphrases the linked sources.