Practice
SEV-1 simulationAPIs and dependencies · Real outage · Reconstructed data · 15–20 minutes
Incident #021 — The dashboard that hammered its own API
Customer impact
The dashboard is mostly unavailable and API errors are rising again. Every API request needs an authorization check from one overloaded service.
| Signal | Current value |
|---|---|
| Dashboard availability | low |
| API availability | falling again |
| Tenant Service | overwhelmed |
| Calls per page view | far above normal |
| CDN and edge traffic | unaffected |
| Last change | patch at 18:58 |
Your task
You are on call for the API platform. Work out why the authorization service cannot keep up, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.
19:00Impact continuing · +0 minabout 0 thousand failed API calls lost since you joined
Investigate · each first look costs 1 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time