Skip to content
All outages

Incident · Medium · Infra

Four good minutes

  • SEV1
  • 12:13 PDT
  • VOICE + VIDEO
90s

The situation

Your chat app's session servers are being resized: bigger pods, fewer of them. In the first zone half the pods stop before handing over, and 17% of all sessions end at once. Clients reconnect and text chat recovers. Voice does not: the cluster that places calls on media servers has stopped answering. At 13:05 you restart all of it. Calls connect, and by 13:09 every node is stuck again.

The evidence

Call-routing requests reaching media serversillustrative
  • requests sent
  • waiting backlog
Logs and signalsReconstructed
12:13session pods, zone b: 50% terminated, no hand-off
12:13call router: backlog growing, requests to media servers ↓
12:43restart call router on one node → recovers, backs up again
13:05restart the whole call-router cluster → some calls connect
13:09all restarted nodes: backlog growing again

Your call

What gets the call-routing cluster to stay up?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last reviewed: Official postmortem