Skip to content
Breaks at Service · kube-proxyCan take the service downVery common

Service has no endpoints

Pods are Running, but the Service in front of them has an empty endpoint list, so connections to it fail.

What you'll seeIllustrative output
$ kubectl describe service web
Name:              web
Namespace:         default
Selector:          app=web-api
Type:              ClusterIP
IP:                10.96.41.7
Port:              http  80/TCP
TargetPort:        8080/TCP
Endpoints:         <none>
$ kubectl get pods --show-labels
NAME                   READY   STATUS    RESTARTS   AGE   LABELS
web-6d5f7c9b8-hx2lm    1/1     Running   0          12m   app=web,pod-template-hash=6d5f7c9b8
web-6d5f7c9b8-tq9wd    1/1     Running   0          12m   app=web,pod-template-hash=6d5f7c9b8
$ kubectl run tmp --rm -it --image=busybox:1.36 --restart=Never -- wget -qO- -T 5 http://web
wget: can't connect to remote host (10.96.41.7): Connection refused

Watch it happen

How Service has no endpoints unfolds

Step 3 · The failure

The selector matches nothing

The Service selects app=web-api. The pods are labelled app=web. The controller finds no match, so the list stays empty. Both pods are green and idle while kube-proxy refuses every connection. The Service lights up: its selector is the fault.

  • refused
  • no match
  • healthy pod

Where it breaks: The Service's selector is what is wrong: it matches no ready pod, so the endpoint list is empty. kube-proxy is the component that then turns connections away, and it is doing its job.

In short

What Service has no endpoints means

A Service does not contain pods. It is a stable virtual IP with a label selector, and a controller keeps a list, held in EndpointSlices, of the ready pods whose labels match that selector. kube-proxy, or the cluster's own dataplane, forwards each connection to one address from the list. If the selector matches nothing, the list is empty. The pods are healthy, the Service exists and resolves in DNS, and every connection still fails, typically with “connection refused” within milliseconds. The usual reasons are a label that does not match the selector after a rename, pods that are not Ready, or a Service in a different namespace from its pods. A close relative gives the same symptom with a full list: the Service's targetPort points at a port the container does not listen on.

In order

Diagnose in 60 seconds

  1. 1
    kubectl describe service <service>
    Look for
    The Endpoints line, the Selector and the TargetPort.
    What it tells you
    Endpoints: <none> means no ready pod matches the selector. A list of addresses rules that out and points at the port or the network instead.
  2. 2
    kubectl get pods -l <selector> -o wide
    Look for
    Use the exact selector from the previous step. Check that pods are listed, and that READY shows them ready.
    What it tells you
    No pods: the labels and the selector disagree. Pods listed but 0/1 Ready: a failing readiness probe keeps them out of the list.
  3. 3
    kubectl get pods --show-labels
    Look for
    The labels the pods really carry, compared with the selector character by character.
    What it tells you
    Shows the typo, the renamed label or the missing label directly.
  4. 4
    kubectl get endpointslices -l kubernetes.io/service-name=<service> -o yaml
    Look for
    The addresses, each one's ready condition, and the port number.
    What it tells you
    Confirms what kube-proxy has been told. The port here must be one the container actually listens on.
  5. 5
    kubectl exec <client-pod> -- wget -qO- -T 5 http://<pod-ip>:<container-port>
    Look for
    Whether a pod answers when called directly, bypassing the Service.
    What it tells you
    Works directly but not through the Service: the Service definition is wrong. Fails directly too: the app, its port or a NetworkPolicy is the problem.

Most common first

Root causes

  1. 1The Service's selector does not match the pods' labels: a typo, a renamed label, or a label changed by a new chart or template.

    Confirm
    kubectl get pods -l with the Service's selector returns nothing, while the pods are plainly running.
    Fix now
    Correct the selector in the Service, or the labels in the pod template, so they agree.
    Prevent
    Generate both from one value in your templates, and check the endpoint count after every deploy.
  2. 2The pods match but are not Ready, so they are left out of the list.

    Confirm
    The pods show 0/1 in the READY column and their events show readiness probe failures.
    Fix now
    Fix what the readiness probe checks, or the probe itself if it tests the wrong port or path.
    Prevent
    Alert on Services that have had no ready endpoints for more than a short time.
  3. 3targetPort does not match the port the container listens on.

    Confirm
    The endpoint list is full, calls to the Service are refused, and calling the pod IP on its real port works.
    Fix now
    Set targetPort to the container's listening port, or to the name of that port.
    Prevent
    Name container ports and refer to them by name in the Service.
  4. 4The Service and the pods are in different namespaces. A selector only ever looks inside the Service's own namespace.

    Confirm
    kubectl get pods -A -l with the selector finds the pods in another namespace.
    Fix now
    Create the Service in the pods' namespace and call it by its full name, service.namespace.
    Prevent
    Keep a workload and its Service in one manifest or chart.
  5. 5The app listens only on localhost inside the container.

    Confirm
    The endpoint list and ports are right, and the app's start-up log says it is bound to 127.0.0.1.
    Fix now
    Bind the server to 0.0.0.0 so it accepts connections on the pod's own address.
    Prevent
    Test the container by calling it from another pod, not only from inside itself.

The tempting wrong fix

Restart the pods

The pods were never the problem: they are Running and would answer if anything reached them. New pods come up with the same labels, the selector still matches none of them, and the endpoint list is just as empty as before.

Before / after

The most common fix: a selector that equals the pod labels

Before
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web-api      # no pod has this label
  ports:
    - port: 80
      targetPort: 8080
After
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web          # same as the pod template's labels
  ports:
    - port: 80
      targetPort: http  # the container port, by name

A Deployment refuses a template whose labels do not match its own selector. A Service has no such check: it accepts any selector and simply ends up with no endpoints.

In the real world

Depends on version and provider

What a client sees when the list is empty depends on what implements Services in your cluster. kube-proxy in its iptables and nftables modes rejects the connection, which shows as “connection refused”. Dataplanes that replace kube-proxy, such as the eBPF ones offered by some CNIs and managed providers, may reject or silently drop instead, and a drop shows as a timeout. An ingress controller or gateway in front of the Service usually turns the same condition into an HTTP 503.

Seen in a real outage

Anything chosen by label breaks quietly when the label goes. At Reddit in 2023, a Kubernetes upgrade removed a node label that the cluster's networking used to pick its route reflectors. Nothing matched any more, and pods lost the ability to reach each other.

Break it: The label that disappeared

Your turn

What would you do?

After a chart upgrade, calls to the “orders” Service fail with “connection refused” in a few milliseconds. All orders pods are Running and 1/1 Ready. What do you check first?

Service has no endpoints questions

Why does my Kubernetes Service have no endpoints?
Because no ready pod matches its selector. Either the pod labels and the selector differ, the pods are in another namespace, or the pods exist but are failing their readiness probe.
What is the difference between port and targetPort in a Service?
port is the port clients use on the Service's own address. targetPort is the port on the pod that the connection is forwarded to. It must be a port the container really listens on.
Why do I get connection refused from a Service when the pods are Running?
Typically the Service has an empty endpoint list, so there is nowhere to forward the connection, or its targetPort points at a port nothing listens on. Running pods do not help if the Service does not select them.

Sources: Kubernetes docs: Debug Services ↗ · Kubernetes docs: Service ↗ · Kubernetes docs: EndpointSlices ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play