Skip to content
Practice

SEV-1 simulationAPIs and dependencies · Real outage · Reconstructed data · 15–20 minutes

Incident #021 — The dashboard that hammered its own API

Customer impact

The dashboard is mostly unavailable and API errors are rising again. Every API request needs an authorization check from one overloaded service.

SignalCurrent value
Dashboard availabilitylow
API availabilityfalling again
Tenant Serviceoverwhelmed
Calls per page viewfar above normal
CDN and edge trafficunaffected
Last changepatch at 18:58

Your task

You are on call for the API platform. Work out why the authorization service cannot keep up, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.

19:00Impact continuing · +0 minabout 0 thousand failed API calls lost since you joined

Investigate · each first look costs 1 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

The patch at 18:58 has made the API worse, and the dashboard is still down. What do you do?