Practice
SEV-1 simulationAPIs and dependencies · Real outage · Reconstructed data · 15–20 minutes
Incident #025 — Fail open or fail closed
Customer impact
Many products are failing at once: identity logins, new device registrations, challenge widgets and dashboard logins. Their shared key-value store cannot reach its central storage.
| Signal | Current value |
|---|---|
| KV requests failing | ≈90% |
| Identity logins | 100% failing |
| Device registrations | failing |
| Dashboard logins | failing |
| Central storage | third-party outage |
| Gateway DNS | mostly working |
Your task
Several product incidents have just been merged under one cause. Work out what they share, using the service map, metrics, logs, traces and recent changes. You cannot fix the provider. Choose how to limit the damage, then name the cause and the permanent fix.
18:08Impact continuing · +0 minabout 0 thousand failed logins and requests lost since you joined
Investigate · each first look costs 2 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time