Skip to content
Practice

SEV-1 simulationCapacity and scaling · Real outage · Reconstructed data · 15–20 minutes

Incident #010 — One server too many

Customer impact

Putting and getting stream records is failing. Metrics, logins and events that depend on the stream service are failing with it.

SignalCurrent value
Put/get record errorselevated
Threads per front-end serverat the OS limit
Shard-map cachesfailing to build
Front-end fleetcapacity added 02:44–03:47
Metrics serviceerrors, alarms blind
Status dashboardcannot post

Your task

You are on call for the stream service's front-end fleet. Work out why servers cannot serve requests, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.

05:20Impact continuing · +0 minabout 0 million failed requests lost since you joined

Investigate · each first look costs 3 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

The stream service is failing and taking other services with it. What do you do first?