Skip to content
Practice

SEV-1 simulationConfig and deploys · Real outage · Reconstructed data · 15–20 minutes

Incident #003 — The network that keeps blinking

Customer impact

The proxy fleet is returning 5xx on and off, in a five-minute rhythm. Your status page is unreachable.

SignalCurrent value
5xx on HTTP trafficon and off, 5 min
Bot feature file≈2× usual size
Requests arrivingnormal
Workers KV errorselevated
Dashboard loginsfailing
Status pageunreachable

Your task

You join the incident call. Work out why the network keeps failing and recovering, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.

11:35Impact continuing · +0 minabout 0 million failed requests lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

Customer traffic is failing every few minutes. What do you do first?