Skip to content
Practice

SEV-1 simulationNetworking and DNS · Real outage · Reconstructed data · 15–20 minutes

Incident #008 — Nobody can find anybody

Customer impact

Product and outfit pages are returning errors. Services in the cluster cannot resolve hostnames.

SignalCurrent value
Store errorshigh, product pages
CoreDNS podsOOMKilled, restarting
DNS queries≈10× normal
Circuit breakersopen
Automatic pagingsilent

Your task

You are the Kubernetes on-call engineer. Another team's on-call has phoned you directly because nothing paged. Find out why name resolution is failing, using the service map, metrics, logs, traces and configuration. Stop the impact first, then name the cause and the permanent fix.

10:30Impact continuing · +0 minabout 0 failed page views lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

The store is failing because nothing in the cluster can resolve names. What do you do first?