Service has no endpoints
Pods are Running, but the Service in front of them has an empty endpoint list, so connections to it fail.
$ kubectl describe service webName: web Namespace: default Selector: app=web-api Type: ClusterIP IP: 10.96.41.7 Port: http 80/TCP TargetPort: 8080/TCP Endpoints: <none>
$ kubectl get pods --show-labelsNAME READY STATUS RESTARTS AGE LABELS web-6d5f7c9b8-hx2lm 1/1 Running 0 12m app=web,pod-template-hash=6d5f7c9b8 web-6d5f7c9b8-tq9wd 1/1 Running 0 12m app=web,pod-template-hash=6d5f7c9b8
$ kubectl run tmp --rm -it --image=busybox:1.36 --restart=Never -- wget -qO- -T 5 http://webwget: can't connect to remote host (10.96.41.7): Connection refused
Watch it happen
How Service has no endpoints unfolds
Step 3 · The failure
The selector matches nothing
The Service selects app=web-api. The pods are labelled app=web. The controller finds no match, so the list stays empty. Both pods are green and idle while kube-proxy refuses every connection. The Service lights up: its selector is the fault.
- refused
- no match
- healthy pod
Step 3 · The failure
The selector matches nothing
The Service selects app=web-api. The pods are labelled app=web. The controller finds no match, so the list stays empty. Both pods are green and idle while kube-proxy refuses every connection. The Service lights up: its selector is the fault.
- refused
- no match
- healthy pod
Where it breaks: The Service's selector is what is wrong: it matches no ready pod, so the endpoint list is empty. kube-proxy is the component that then turns connections away, and it is doing its job.
In short
What Service has no endpoints means
A Service does not contain pods. It is a stable virtual IP with a label selector, and a controller keeps a list, held in EndpointSlices, of the ready pods whose labels match that selector. kube-proxy, or the cluster's own dataplane, forwards each connection to one address from the list. If the selector matches nothing, the list is empty. The pods are healthy, the Service exists and resolves in DNS, and every connection still fails, typically with “connection refused” within milliseconds. The usual reasons are a label that does not match the selector after a rename, pods that are not Ready, or a Service in a different namespace from its pods. A close relative gives the same symptom with a full list: the Service's targetPort points at a port the container does not listen on.
In order
Diagnose in 60 seconds
- 1
kubectl describe service <service>- Look for
- The Endpoints line, the Selector and the TargetPort.
- What it tells you
- Endpoints: <none> means no ready pod matches the selector. A list of addresses rules that out and points at the port or the network instead.
- 2
kubectl get pods -l <selector> -o wide- Look for
- Use the exact selector from the previous step. Check that pods are listed, and that READY shows them ready.
- What it tells you
- No pods: the labels and the selector disagree. Pods listed but 0/1 Ready: a failing readiness probe keeps them out of the list.
- 3
kubectl get pods --show-labels- Look for
- The labels the pods really carry, compared with the selector character by character.
- What it tells you
- Shows the typo, the renamed label or the missing label directly.
- 4
kubectl get endpointslices -l kubernetes.io/service-name=<service> -o yaml- Look for
- The addresses, each one's ready condition, and the port number.
- What it tells you
- Confirms what kube-proxy has been told. The port here must be one the container actually listens on.
- 5
kubectl exec <client-pod> -- wget -qO- -T 5 http://<pod-ip>:<container-port>- Look for
- Whether a pod answers when called directly, bypassing the Service.
- What it tells you
- Works directly but not through the Service: the Service definition is wrong. Fails directly too: the app, its port or a NetworkPolicy is the problem.
Most common first
Root causes
1The Service's selector does not match the pods' labels: a typo, a renamed label, or a label changed by a new chart or template.
- Confirm
- kubectl get pods -l with the Service's selector returns nothing, while the pods are plainly running.
- Fix now
- Correct the selector in the Service, or the labels in the pod template, so they agree.
- Prevent
- Generate both from one value in your templates, and check the endpoint count after every deploy.
2The pods match but are not Ready, so they are left out of the list.
- Confirm
- The pods show 0/1 in the READY column and their events show readiness probe failures.
- Fix now
- Fix what the readiness probe checks, or the probe itself if it tests the wrong port or path.
- Prevent
- Alert on Services that have had no ready endpoints for more than a short time.
3targetPort does not match the port the container listens on.
- Confirm
- The endpoint list is full, calls to the Service are refused, and calling the pod IP on its real port works.
- Fix now
- Set targetPort to the container's listening port, or to the name of that port.
- Prevent
- Name container ports and refer to them by name in the Service.
4The Service and the pods are in different namespaces. A selector only ever looks inside the Service's own namespace.
- Confirm
- kubectl get pods -A -l with the selector finds the pods in another namespace.
- Fix now
- Create the Service in the pods' namespace and call it by its full name, service.namespace.
- Prevent
- Keep a workload and its Service in one manifest or chart.
5The app listens only on localhost inside the container.
- Confirm
- The endpoint list and ports are right, and the app's start-up log says it is bound to 127.0.0.1.
- Fix now
- Bind the server to 0.0.0.0 so it accepts connections on the pod's own address.
- Prevent
- Test the container by calling it from another pod, not only from inside itself.
The tempting wrong fix
Restart the pods
Before / after
The most common fix: a selector that equals the pod labels
apiVersion: v1
kind: Service
metadata:
name: web
spec:
selector:
app: web-api # no pod has this label
ports:
- port: 80
targetPort: 8080apiVersion: v1
kind: Service
metadata:
name: web
spec:
selector:
app: web # same as the pod template's labels
ports:
- port: 80
targetPort: http # the container port, by nameA Deployment refuses a template whose labels do not match its own selector. A Service has no such check: it accepts any selector and simply ends up with no endpoints.
In the real world
Depends on version and provider
Seen in a real outage
Anything chosen by label breaks quietly when the label goes. At Reddit in 2023, a Kubernetes upgrade removed a node label that the cluster's networking used to pick its route reflectors. Nothing matched any more, and pods lost the ability to reach each other.
Break it: The label that disappearedYour turn
What would you do?
After a chart upgrade, calls to the “orders” Service fail with “connection refused” in a few milliseconds. All orders pods are Running and 1/1 Ready. What do you check first?
Related errors
Service has no endpoints questions
- Why does my Kubernetes Service have no endpoints?
- Because no ready pod matches its selector. Either the pod labels and the selector differ, the pods are in another namespace, or the pods exist but are failing their readiness probe.
- What is the difference between port and targetPort in a Service?
- port is the port clients use on the Service's own address. targetPort is the port on the pod that the connection is forwarded to. It must be a port the container really listens on.
- Why do I get connection refused from a Service when the pods are Running?
- Typically the Service has an empty endpoint list, so there is nowhere to forward the connection, or its targetPort points at a port nothing listens on. Running pods do not help if the Service does not select them.
Sources: Kubernetes docs: Debug Services ↗ · Kubernetes docs: Service ↗ · Kubernetes docs: EndpointSlices ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.