Practice
SEV-1 simulationCapacity and scaling · Real outage · Reconstructed data · 15–20 minutes
Incident #020 — Smaller to save money
Customer impact
The dashboard has been down since 17:21. Build agents are still running builds, but nobody can see or manage them.
| Signal | Current value |
|---|---|
| Dashboard | down |
| Database CPU | 100% |
| Dashboard servers | cycling up and down |
| EC2 API | request limit exceeded |
| Build agents | still running builds |
| AWS console | errors (IAM) |
Your task
It is 21:00 and you have just woken to the pages. Work out why the dashboard is down and why it is not recovering, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.
21:00Impact continuing · +0 minabout 0 customers blocked lost since you joined
Investigate · each first look costs 3 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time