Skip to content

Real-time & chat · 4 of 6

Step 1 · The pattern

Gateways hold the connections

Every client keeps one connection open to a gateway. One dot stands for a thousand clients, so each gateway holds 16k. New connections need a handshake, and a gateway can run only ten at once: that is the hs count.

  • connecting
  • connected

In short

Reconnect storms after a deploy: how it works and how it breaks

A gateway server for a real-time app holds tens of thousands of open connections. They cost little to keep open and a lot to set up: each needs a handshake and a session resume. Restart a gateway and every one of its connections drops in the same instant, and every client reconnects in the same instant to the gateways that remain. One restart may fit. Restart several close together and the rest of the fleet meets a wall of reconnects: connects fail, clients retry, and a surviving gateway can fall over and add its own clients to the wave. So a routine deploy needs a drain. Take the gateway out of rotation, close its connections a few at a time or tell clients to move, and stop it only when it is empty. Then wait for reconnects to settle before touching the next gateway.

How this comes up in interviews

"Each of your gateways holds tens of thousands of WebSockets. How do you deploy a new version without clients seeing failed connects?"

Then they ask

  1. Your rolling deploy waits for the new process to pass its health check before the next restart. Why do connects still fail?
  2. How do you decide how fast a gateway may drain, and what signal gates the next step?
  3. The orchestrator kills a process after a fixed grace period. How does that interact with a slow drain?

A strong answer covers

  • Open connections cost little to keep and a lot to set up: each needs a handshake and a session resume.
  • A restart drops every connection on a gateway in the same instant, and all those clients reconnect at once to the gateways that remain.
  • Restarts close together stack the waves: connects fail, clients retry, and a surviving gateway can fall over and add its own clients.
  • Drain first: take the gateway out of rotation, close connections a few at a time or tell clients to move, and stop it only when empty.
  • Wait for reconnects to settle before touching the next gateway. A passing health check does not tell you that.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Official postmortem

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.