Skip to content
Practice

SEV-1 simulationCapacity and scaling · Real outage · Reconstructed data · 15–20 minutes

Incident #020 — Smaller to save money

Customer impact

The dashboard has been down since 17:21. Build agents are still running builds, but nobody can see or manage them.

SignalCurrent value
Dashboarddown
Database CPU100%
Dashboard serverscycling up and down
EC2 APIrequest limit exceeded
Build agentsstill running builds
AWS consoleerrors (IAM)

Your task

It is 21:00 and you have just woken to the pages. Work out why the dashboard is down and why it is not recovering, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.

21:00Impact continuing · +0 minabout 0 customers blocked lost since you joined

Investigate · each first look costs 3 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

The dashboard has been down for hours. What do you do?