Skip to content
Practice

SEV-1 simulationAPIs and dependencies · Real outage · Reconstructed data · 15–20 minutes

Incident #025 — Fail open or fail closed

Customer impact

Many products are failing at once: identity logins, new device registrations, challenge widgets and dashboard logins. Their shared key-value store cannot reach its central storage.

SignalCurrent value
KV requests failing≈90%
Identity logins100% failing
Device registrationsfailing
Dashboard loginsfailing
Central storagethird-party outage
Gateway DNSmostly working

Your task

Several product incidents have just been merged under one cause. Work out what they share, using the service map, metrics, logs, traces and recent changes. You cannot fix the provider. Choose how to limit the damage, then name the cause and the permanent fix.

18:08Impact continuing · +0 minabout 0 thousand failed logins and requests lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

The storage provider is down and you cannot fix it. What do you do?