Skip to content
Breaks at CoreDNS · Your containerCan take the service downCommon

DNS resolution failure

Pods cannot turn names into addresses, or only slowly, because queries to CoreDNS fail or time out.

What you'll seeIllustrative output
$ kubectl exec -it web-6d5f7c9b8-hx2lm -- nslookup api.example.com
;; communications error to 10.96.0.10#53: timed out
;; communications error to 10.96.0.10#53: timed out
;; no servers could be reached
$ kubectl exec web-6d5f7c9b8-hx2lm -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
$ kubectl logs -n kube-system -l k8s-app=kube-dns --tail=4
[INFO] 10.244.1.17:41233 - 5121 "A IN api.example.com.default.svc.cluster.local. udp 59 false 512" NXDOMAIN qr,aa,rd 152 0.000131s
[INFO] 10.244.1.17:52710 - 5122 "A IN api.example.com.svc.cluster.local. udp 51 false 512" NXDOMAIN qr,aa,rd 144 0.000098s
[INFO] 10.244.1.17:38164 - 5123 "A IN api.example.com.cluster.local. udp 47 false 512" NXDOMAIN qr,aa,rd 140 0.000087s
[INFO] 10.244.1.17:49002 - 5124 "A IN api.example.com. udp 33 false 512" NOERROR qr,rd,ra 64 0.004211s

Watch it happen

How DNS resolution failure unfolds

Step 3 · The failure

CoreDNS runs out of capacity

Every pod now calls external names at volume, and three of every four queries are wasted. CoreDNS fills up, its queue overflows and new queries are lost. The client waits out its timeout, 5 seconds by default, and then reports that the name could not be resolved.

  • wasted query
  • answered
  • lost

Where it breaks: CoreDNS is where queries pile up and time out. The pod's own resolver settings are often what overloads it: the search list and ndots:5 turn each external lookup into several queries.

In short

What DNS resolution failure means

Inside a cluster, every name lookup goes to CoreDNS. The kubelet writes each pod's resolv.conf with the DNS Service's address, a search list of cluster suffixes, and the option ndots:5. That option means any name with fewer than five dots is first tried with each suffix appended, and only then as written. For an in-cluster name this is what makes short names work. For an external name such as api.example.com it means three queries that can only fail before the one that succeeds, and twice that when the client asks for IPv6 addresses too. At volume, this multiplies the load on CoreDNS until queries queue and time out, and applications report resolution failures or five second pauses. The other common causes are a NetworkPolicy that blocks port 53, CoreDNS pods that are down, and a failing upstream resolver.

In order

Diagnose in 60 seconds

  1. 1
    kubectl exec -it <pod> -- nslookup kubernetes.default
    Look for
    Whether an in-cluster name resolves, and how long it takes. If the image has no nslookup, run a temporary pod that does.
    What it tells you
    Fails: the pod cannot reach CoreDNS, or CoreDNS is down. Works while external names fail: look at the upstream resolver or at query volume.
  2. 2
    kubectl exec <pod> -- cat /etc/resolv.conf
    Look for
    The nameserver address, the search list and the ndots option.
    What it tells you
    The nameserver should be the cluster DNS Service's IP. A different address means the pod's dnsPolicy is not the cluster default.
  3. 3
    kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
    Look for
    That the CoreDNS pods are Running and Ready, and their restart counts.
    What it tells you
    Rules in a CoreDNS outage: crash loops, pods Pending, or all replicas on one failed node.
  4. 4
    kubectl get endpointslices -n kube-system -l kubernetes.io/service-name=kube-dns
    Look for
    That the DNS Service has endpoints.
    What it tells you
    No endpoints means queries to the Service address go nowhere, whatever the pods look like.
  5. 5
    kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
    Look for
    Errors such as timeouts to the upstream server or a detected loop. With the log plugin enabled, the stream of NXDOMAIN answers for names ending in cluster.local.
    What it tells you
    Upstream timeouts point outside the cluster. A flood of NXDOMAIN for external names with cluster suffixes is the search list at work.
  6. 6
    kubectl get networkpolicy -n <namespace>
    Look for
    Any policy with Egress in its policy types that selects the failing pods.
    What it tells you
    An egress policy without a rule for port 53 blocks every lookup. See the NetworkPolicy page.

Most common first

Root causes

  1. 1CoreDNS is overloaded: too few replicas for the query volume, made worse by search-list expansion of external names.

    Confirm
    CoreDNS pods sit at their CPU limit, failures rise with traffic, and its log is dominated by NXDOMAIN for names with cluster suffixes.
    Fix now
    Add CoreDNS replicas or CPU. Cut the wasted queries with fully qualified names or a lower ndots for the busiest workloads.
    Prevent
    Scale CoreDNS with cluster size, and consider a node-local DNS cache.
  2. 2A NetworkPolicy isolates the pods for egress and has no rule allowing DNS.

    Confirm
    Lookups fail only from pods the policy selects, and work from an unselected pod in the same namespace.
    Fix now
    Add an egress rule for UDP and TCP port 53 to the cluster DNS pods.
    Prevent
    Include the DNS rule in every egress policy template.
  3. 3CoreDNS is not running: crash-looping, unschedulable, or all replicas lost with one node.

    Confirm
    The kube-dns pods are not Ready, or the DNS Service has no endpoints.
    Fix now
    Fix what stops the pods, starting with their logs and events.
    Prevent
    Run at least two replicas spread across nodes, with a PodDisruptionBudget.
  4. 4The upstream resolver that CoreDNS forwards to is slow or unreachable.

    Confirm
    In-cluster names resolve at once while external names time out, and CoreDNS logs timeouts to its forward address.
    Fix now
    Restore the upstream resolver, or point CoreDNS's forward setting at one that works.
    Prevent
    Forward to more than one upstream, and keep caching enabled.
  5. 5Lost UDP packets on the node: occasional lookups hang for the resolver's full timeout and then succeed.

    Confirm
    Isolated delays of about five seconds with a healthy, idle CoreDNS.
    Fix now
    Use a node-local DNS cache, which avoids the connection-tracking step where these packets are typically lost.
    Prevent
    Keep nodes' kernels and the CNI current, and monitor DNS latency from inside pods.

The tempting wrong fix

Restart CoreDNS

A restart empties CoreDNS's cache, so the first thing the new pods face is the same flood of queries with nothing cached, and for a moment there are fewer replicas to take it. If a policy or the upstream is the cause, CoreDNS was never broken. Find out which of those it is first.

Before / after

The most common fix: fewer wasted queries per lookup

Before
spec:
  containers:
    - name: web
      env:
        - name: PAYMENTS_URL
          value: https://api.example.com   # 2 dots: tried with 3 suffixes first
After
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"                          # names with 2+ dots are asked as written first
  containers:
    - name: web
      env:
        - name: PAYMENTS_URL
          value: https://api.example.com

With ndots:2, short in-cluster names such as web or web.default still go through the search list and keep working. A trailing dot on a single name, api.example.com., has the same effect for that name alone.

In the real world

Depends on version and provider

The search list and ndots:5 are what the kubelet writes for the default ClusterFirst DNS policy; pods with another dnsPolicy or their own dnsConfig differ. The five second timeout and the number of retries belong to the resolver library in the container image, not to Kubernetes, and some images behave differently. Managed clusters do not all run CoreDNS the same way: some add a node-local cache by default, in which case the nameserver is a link-local address on the node, and some use the cloud provider's own DNS in place of CoreDNS.

Seen in a real outage

Overloaded DNS servers fail gradually, because clients keep working from cached answers until those expire. At Square in 2023, changes that overloaded internal DNS took an afternoon to turn into a full outage, and the tools needed to fix it depended on DNS too.

Break it: Failing slowly, then all at once

Your turn

What would you do?

At peak, calls from pods to api.partner.com sometimes take five seconds or fail to resolve. In-cluster names are fine. CoreDNS is at its CPU limit and its log is full of NXDOMAIN for api.partner.com.default.svc.cluster.local. What is going on?

DNS resolution failure questions

What does ndots:5 mean in a Kubernetes pod?
It tells the resolver that a name with fewer than five dots should first be tried with each suffix from the search list, and only then as written. It makes short Service names work, and makes external names cost several extra queries.
How do I test DNS from inside a pod?
Run a lookup with kubectl exec, for example nslookup kubernetes.default. If the image has no DNS tools, start a temporary pod from an image that includes them and test from there.
Why are DNS lookups in my cluster slow by exactly five seconds?
Five seconds is the usual resolver timeout. A query or its reply was lost, the client waited, then retried successfully. Typical reasons are an overloaded CoreDNS or UDP packets dropped on the node.

Sources: Kubernetes docs: DNS for Services and Pods ↗ · Kubernetes docs: Debugging DNS resolution ↗ · Kubernetes docs: Using NodeLocal DNSCache ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play