Skip to content
Practice

SEV-2 simulationTime and limits · Real outage · Reconstructed data · 15–20 minutes

Incident #018 — Time went backwards

Customer impact

About 0.2% of DNS queries are failing, all for CNAME records, and some HTTP requests fail with them.

SignalCurrent value
DNS queries failing≈0.2%
HTTP requests with errorsunder 1%
Records affectedCNAME only
DNS server panicsrecovered, repeating
System clockleap second at midnight

Your task

You have just been escalated to. Work out why CNAME lookups fail, using the service map, metrics, logs, traces and the code. Stop the impact first, then name the cause and the permanent fix.

00:12Impact continuing · +0 minabout 0 thousand failed lookups lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

CNAME lookups are failing across many data centers. What do you do?