Practice
SEV-1 simulationConfig and deploys · Real outage · Reconstructed data · 15–20 minutes
Incident #004 — Crash-looping everywhere at once
Customer impact
API requests are failing in every region. The quota-check service is crash-looping everywhere.
| Signal | Current value |
|---|---|
| API error rate | high, every region |
| quota-check tasks | crash-looping |
| Last release | 29 May |
| Policy DB us-central1 | load rising |
| Status page | down |
Your task
You are on call for quota-check, the service every API request passes through. Find out why it is crashing everywhere, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.
10:53Impact continuing · +0 minabout 0 million failed API calls lost since you joined
Investigate · each first look costs 2 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time