Practice
SEV-2 simulationTime and limits · Real outage · Reconstructed data · 15–20 minutes
Incident #018 — Time went backwards
Customer impact
About 0.2% of DNS queries are failing, all for CNAME records, and some HTTP requests fail with them.
| Signal | Current value |
|---|---|
| DNS queries failing | ≈0.2% |
| HTTP requests with errors | under 1% |
| Records affected | CNAME only |
| DNS server panics | recovered, repeating |
| System clock | leap second at midnight |
Your task
You have just been escalated to. Work out why CNAME lookups fail, using the service map, metrics, logs, traces and the code. Stop the impact first, then name the cause and the permanent fix.
00:12Impact continuing · +0 minabout 0 thousand failed lookups lost since you joined
Investigate · each first look costs 2 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time