DNS resolution failure
Pods cannot turn names into addresses, or only slowly, because queries to CoreDNS fail or time out.
$ kubectl exec -it web-6d5f7c9b8-hx2lm -- nslookup api.example.com;; communications error to 10.96.0.10#53: timed out ;; communications error to 10.96.0.10#53: timed out ;; no servers could be reached
$ kubectl exec web-6d5f7c9b8-hx2lm -- cat /etc/resolv.confsearch default.svc.cluster.local svc.cluster.local cluster.local nameserver 10.96.0.10 options ndots:5
$ kubectl logs -n kube-system -l k8s-app=kube-dns --tail=4[INFO] 10.244.1.17:41233 - 5121 "A IN api.example.com.default.svc.cluster.local. udp 59 false 512" NXDOMAIN qr,aa,rd 152 0.000131s [INFO] 10.244.1.17:52710 - 5122 "A IN api.example.com.svc.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000098s [INFO] 10.244.1.17:38164 - 5123 "A IN api.example.com.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 140 0.000087s [INFO] 10.244.1.17:49002 - 5124 "A IN api.example.com. udp 33 false 512" NOERROR qr,rd,ra 64 0.004211s
Watch it happen
How DNS resolution failure unfolds
Step 3 · The failure
CoreDNS runs out of capacity
Every pod now calls external names at volume, and three of every four queries are wasted. CoreDNS fills up, its queue overflows and new queries are lost. The client waits out its timeout, 5 seconds by default, and then reports that the name could not be resolved.
- wasted query
- answered
- lost
Step 3 · The failure
CoreDNS runs out of capacity
Every pod now calls external names at volume, and three of every four queries are wasted. CoreDNS fills up, its queue overflows and new queries are lost. The client waits out its timeout, 5 seconds by default, and then reports that the name could not be resolved.
- wasted query
- answered
- lost
Where it breaks: CoreDNS is where queries pile up and time out. The pod's own resolver settings are often what overloads it: the search list and ndots:5 turn each external lookup into several queries.
In short
What DNS resolution failure means
Inside a cluster, every name lookup goes to CoreDNS. The kubelet writes each pod's resolv.conf with the DNS Service's address, a search list of cluster suffixes, and the option ndots:5. That option means any name with fewer than five dots is first tried with each suffix appended, and only then as written. For an in-cluster name this is what makes short names work. For an external name such as api.example.com it means three queries that can only fail before the one that succeeds, and twice that when the client asks for IPv6 addresses too. At volume, this multiplies the load on CoreDNS until queries queue and time out, and applications report resolution failures or five second pauses. The other common causes are a NetworkPolicy that blocks port 53, CoreDNS pods that are down, and a failing upstream resolver.
In order
Diagnose in 60 seconds
- 1
kubectl exec -it <pod> -- nslookup kubernetes.default- Look for
- Whether an in-cluster name resolves, and how long it takes. If the image has no nslookup, run a temporary pod that does.
- What it tells you
- Fails: the pod cannot reach CoreDNS, or CoreDNS is down. Works while external names fail: look at the upstream resolver or at query volume.
- 2
kubectl exec <pod> -- cat /etc/resolv.conf- Look for
- The nameserver address, the search list and the ndots option.
- What it tells you
- The nameserver should be the cluster DNS Service's IP. A different address means the pod's dnsPolicy is not the cluster default.
- 3
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide- Look for
- That the CoreDNS pods are Running and Ready, and their restart counts.
- What it tells you
- Rules in a CoreDNS outage: crash loops, pods Pending, or all replicas on one failed node.
- 4
kubectl get endpointslices -n kube-system -l kubernetes.io/service-name=kube-dns- Look for
- That the DNS Service has endpoints.
- What it tells you
- No endpoints means queries to the Service address go nowhere, whatever the pods look like.
- 5
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50- Look for
- Errors such as timeouts to the upstream server or a detected loop. With the log plugin enabled, the stream of NXDOMAIN answers for names ending in cluster.local.
- What it tells you
- Upstream timeouts point outside the cluster. A flood of NXDOMAIN for external names with cluster suffixes is the search list at work.
- 6
kubectl get networkpolicy -n <namespace>- Look for
- Any policy with Egress in its policy types that selects the failing pods.
- What it tells you
- An egress policy without a rule for port 53 blocks every lookup. See the NetworkPolicy page.
Most common first
Root causes
1CoreDNS is overloaded: too few replicas for the query volume, made worse by search-list expansion of external names.
- Confirm
- CoreDNS pods sit at their CPU limit, failures rise with traffic, and its log is dominated by NXDOMAIN for names with cluster suffixes.
- Fix now
- Add CoreDNS replicas or CPU. Cut the wasted queries with fully qualified names or a lower ndots for the busiest workloads.
- Prevent
- Scale CoreDNS with cluster size, and consider a node-local DNS cache.
2A NetworkPolicy isolates the pods for egress and has no rule allowing DNS.
- Confirm
- Lookups fail only from pods the policy selects, and work from an unselected pod in the same namespace.
- Fix now
- Add an egress rule for UDP and TCP port 53 to the cluster DNS pods.
- Prevent
- Include the DNS rule in every egress policy template.
3CoreDNS is not running: crash-looping, unschedulable, or all replicas lost with one node.
- Confirm
- The kube-dns pods are not Ready, or the DNS Service has no endpoints.
- Fix now
- Fix what stops the pods, starting with their logs and events.
- Prevent
- Run at least two replicas spread across nodes, with a PodDisruptionBudget.
4The upstream resolver that CoreDNS forwards to is slow or unreachable.
- Confirm
- In-cluster names resolve at once while external names time out, and CoreDNS logs timeouts to its forward address.
- Fix now
- Restore the upstream resolver, or point CoreDNS's forward setting at one that works.
- Prevent
- Forward to more than one upstream, and keep caching enabled.
5Lost UDP packets on the node: occasional lookups hang for the resolver's full timeout and then succeed.
- Confirm
- Isolated delays of about five seconds with a healthy, idle CoreDNS.
- Fix now
- Use a node-local DNS cache, which avoids the connection-tracking step where these packets are typically lost.
- Prevent
- Keep nodes' kernels and the CNI current, and monitor DNS latency from inside pods.
The tempting wrong fix
Restart CoreDNS
Before / after
The most common fix: fewer wasted queries per lookup
spec:
containers:
- name: web
env:
- name: PAYMENTS_URL
value: https://api.example.com # 2 dots: tried with 3 suffixes firstspec:
dnsConfig:
options:
- name: ndots
value: "2" # names with 2+ dots are asked as written first
containers:
- name: web
env:
- name: PAYMENTS_URL
value: https://api.example.comWith ndots:2, short in-cluster names such as web or web.default still go through the search list and keep working. A trailing dot on a single name, api.example.com., has the same effect for that name alone.
In the real world
Depends on version and provider
Seen in a real outage
Overloaded DNS servers fail gradually, because clients keep working from cached answers until those expire. At Square in 2023, changes that overloaded internal DNS took an afternoon to turn into a full outage, and the tools needed to fix it depended on DNS too.
Break it: Failing slowly, then all at onceYour turn
What would you do?
At peak, calls from pods to api.partner.com sometimes take five seconds or fail to resolve. In-cluster names are fine. CoreDNS is at its CPU limit and its log is full of NXDOMAIN for api.partner.com.default.svc.cluster.local. What is going on?
Related errors
DNS resolution failure questions
- What does ndots:5 mean in a Kubernetes pod?
- It tells the resolver that a name with fewer than five dots should first be tried with each suffix from the search list, and only then as written. It makes short Service names work, and makes external names cost several extra queries.
- How do I test DNS from inside a pod?
- Run a lookup with kubectl exec, for example nslookup kubernetes.default. If the image has no DNS tools, start a temporary pod from an image that includes them and test from there.
- Why are DNS lookups in my cluster slow by exactly five seconds?
- Five seconds is the usual resolver timeout. A query or its reply was lost, the client waited, then retried successfully. Typical reasons are an overloaded CoreDNS or UDP packets dropped on the node.
Sources: Kubernetes docs: DNS for Services and Pods ↗ · Kubernetes docs: Debugging DNS resolution ↗ · Kubernetes docs: Using NodeLocal DNSCache ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.