Skip to content
Practice

SEV-1 simulationAPIs and dependencies · Real outage · Reconstructed data · 15–20 minutes

Incident #024 — The dependency that never said no

Customer impact

About a third of user connections have stopped delivering messages. The servers that hold those connections are running out of memory and crashing.

SignalCurrent value
Connections not delivering≈⅓
Session serversout of memory, crashing
Presence clusterone node lost
Reconnects to presencemillions at once

Your task

You are on call for real-time messaging. Work out why one misbehaving server is taking down the session tier, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.

12:00Impact continuing · +0 minabout 0 thousand users without messages lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

A third of users are not getting messages, and session servers keep crashing. What do you do first?