Skip to content
Practice

SEV-1 simulationAPIs and dependencies · Real outage · Reconstructed data · 15–20 minutes

Incident #023 — The backlog that came back at once

Customer impact

Builds are barely starting, about one a minute instead of many a second. The build queue keeps growing and the main database is unresponsive.

SignalCurrent value
Builds starting≈1 a minute
Build queuegrowing
Main databasetiming out
Incoming push hooksseveral × normal peak
New database hardwarehours to sync

Your task

Several fixes have already been tried today. Work out why the database cannot recover, using the service map, metrics, logs, traces and recent changes. Choose what to do now, then name the cause and the permanent fix.

16:30Impact continuing · +0 minabout 0 builds waiting lost since you joined

Investigate · each first look costs 3 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

The database is buried and several fixes have failed. What do you do now?