Practice
SEV-1 simulationNetworking and DNS · Real outage · Reconstructed data · 15–20 minutes
Incident #008 — Nobody can find anybody
Customer impact
Product and outfit pages are returning errors. Services in the cluster cannot resolve hostnames.
| Signal | Current value |
|---|---|
| Store errors | high, product pages |
| CoreDNS pods | OOMKilled, restarting |
| DNS queries | ≈10× normal |
| Circuit breakers | open |
| Automatic paging | silent |
Your task
You are the Kubernetes on-call engineer. Another team's on-call has phoned you directly because nothing paged. Find out why name resolution is failing, using the service map, metrics, logs, traces and configuration. Stop the impact first, then name the cause and the permanent fix.
10:30Impact continuing · +0 minabout 0 failed page views lost since you joined
Investigate · each first look costs 2 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time