Skip to content
Practice

SEV-1 simulationDatabases · Real outage · Reconstructed data · 15–20 minutes

Incident #006 — Forty-three seconds

Customer impact

The site is slow and some pages show out-of-date data. Database primaries have moved to the other coast.

SignalCurrent value
Web p95≈2.4 seconds
MySQL primariesWest Coast
Writes only on East954, busiest cluster
Network linkback after 43 s
Webhooks and Pagesrunning
Deployslocked

Your task

You are the incident commander. Work out why a 43-second network blip left the site degraded, using the service map, metrics, logs, traces and recent changes. Then choose what to do first, and name the cause and the permanent fix.

23:13Impact continuing · +0 minabout 0 thousand writes at risk lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

The site is degraded and the two coasts disagree about the data. What do you do first?