Practice
SEV-1 simulationNetworking and DNS · Real outage · Reconstructed data · 15–20 minutes
Incident #011 — The database that vanished
Customer impact
Clients cannot connect to the regional database endpoint. Services that store their state in it, including instance launches, are failing.
| Signal | Current value |
|---|---|
| Database API | unreachable |
| Endpoint DNS | no addresses |
| Load balancers | healthy |
| DNS automation | stuck |
| Instance launches | failing |
| Request routers | idle |
Your task
You are on call for the database's DNS management. Work out why the endpoint has no addresses, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.
00:40Impact continuing · +0 minabout 0 million failed requests lost since you joined
Investigate · each first look costs 2 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time