Practice
SEV-1 simulationAPIs and dependencies · Real outage · Reconstructed data · 15–20 minutes
Incident #024 — The dependency that never said no
Customer impact
About a third of user connections have stopped delivering messages. The servers that hold those connections are running out of memory and crashing.
| Signal | Current value |
|---|---|
| Connections not delivering | ≈⅓ |
| Session servers | out of memory, crashing |
| Presence cluster | one node lost |
| Reconnects to presence | millions at once |
Your task
You are on call for real-time messaging. Work out why one misbehaving server is taking down the session tier, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.
12:00Impact continuing · +0 minabout 0 thousand users without messages lost since you joined
Investigate · each first look costs 2 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time