Skip to content
Practice

SEV-1 simulationConfig and deploys · Real outage · Reconstructed data · 15–20 minutes

Incident #004 — Crash-looping everywhere at once

Customer impact

API requests are failing in every region. The quota-check service is crash-looping everywhere.

SignalCurrent value
API error ratehigh, every region
quota-check taskscrash-looping
Last release29 May
Policy DB us-central1load rising
Status pagedown

Your task

You are on call for quota-check, the service every API request passes through. Find out why it is crashing everywhere, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.

10:53Impact continuing · +0 minabout 0 million failed API calls lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

API requests are failing in every region. What do you do first?