Practice
SEV-1 simulationCapacity and scaling · Real outage · Reconstructed data · 15–20 minutes
Incident #010 — One server too many
Customer impact
Putting and getting stream records is failing. Metrics, logins and events that depend on the stream service are failing with it.
| Signal | Current value |
|---|---|
| Put/get record errors | elevated |
| Threads per front-end server | at the OS limit |
| Shard-map caches | failing to build |
| Front-end fleet | capacity added 02:44–03:47 |
| Metrics service | errors, alarms blind |
| Status dashboard | cannot post |
Your task
You are on call for the stream service's front-end fleet. Work out why servers cannot serve requests, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.
05:20Impact continuing · +0 minabout 0 million failed requests lost since you joined
Investigate · each first look costs 3 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time