Incident · Medium · Infra
Four good minutes
- SEV1
- 12:13 PDT
- VOICE + VIDEO
90s
The situation
Your chat app's session servers are being resized: bigger pods, fewer of them. In the first zone half the pods stop before handing over, and 17% of all sessions end at once. Clients reconnect and text chat recovers. Voice does not: the cluster that places calls on media servers has stopped answering. At 13:05 you restart all of it. Calls connect, and by 13:09 every node is stuck again.
The evidence
- requests sent
- waiting backlog
Logs and signalsReconstructed
12:13session pods, zone b: 50% terminated, no hand-off12:13call router: backlog growing, requests to media servers ↓12:43restart call router on one node → recovers, backs up again13:05restart the whole call-router cluster → some calls connect13:09all restarted nodes: backlog growing again
Your call
What gets the call-routing cluster to stay up?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.