Skip to content
Practice

SEV-1 simulationNetworking and DNS · Real outage · Reconstructed data · 15–20 minutes

Incident #011 — The database that vanished

Customer impact

Clients cannot connect to the regional database endpoint. Services that store their state in it, including instance launches, are failing.

SignalCurrent value
Database APIunreachable
Endpoint DNSno addresses
Load balancershealthy
DNS automationstuck
Instance launchesfailing
Request routersidle

Your task

You are on call for the database's DNS management. Work out why the endpoint has no addresses, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.

00:40Impact continuing · +0 minabout 0 million failed requests lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

Nobody can reach the database and other services are failing with it. What do you do first?