Skip to content
Breaks at Your container · ServiceCan take the service downVery common

Readiness probe failure

The pod is Running but not Ready, so the Service has stopped sending it traffic. The container is not restarted.

What you'll seeIllustrative output
$ kubectl get pods -l app=web
NAME                   READY   STATUS    RESTARTS   AGE
web-6b7f9c5d8-4hx2m    0/1     Running   0          12m
web-6b7f9c5d8-9qk7t    0/1     Running   0          12m
web-6b7f9c5d8-tz5wd    0/1     Running   0          12m
$ kubectl describe pod web-6b7f9c5d8-4hx2m
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Events:
  Type     Reason     Age                 From     Message
  Warning  Unhealthy  3s (x41 over 6m)    kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503
$ kubectl get endpointslices -l kubernetes.io/service-name=web
NAME        ADDRESSTYPE   PORTS   ENDPOINTS                          AGE
web-x7k2p   IPv4          8080    10.42.1.17,10.42.2.9,10.42.3.21   12m

Watch it happen

How Readiness probe failure unfolds

Step 3 · The failure

Every pod fails at once

All three checks depend on the same thing, and it is down. One by one the pods are pulled out until the Service has no endpoints left. Every request now fails, although all three containers are still Running.

  • failed
  • NotReady

Where it breaks: Your container fails its own readiness check. The kubelet reports that, and the EndpointSlice controller removes the pod from the Service. Nothing is broken in Kubernetes: it is doing what the probe told it to.

In short

What Readiness probe failure means

A readiness probe asks one question: can this container take requests right now? While the answer is no, the pod stays Running but shows 0/1 in the READY column. The kubelet records the failure, the pod's Ready condition turns False, and the EndpointSlice controller marks the pod as not ready in the Service's endpoints, so the proxy stops routing to it. Nobody restarts the container; restarting is the job of the liveness probe. With one pod out, the others carry the traffic and users notice nothing. The outage comes when every pod fails at once, typically because the readiness endpoint checks a shared dependency or because the probe points at the wrong port or path. Then the Service has no endpoints, and clients get connection errors or a 503 from the ingress in front.

In order

Diagnose in 60 seconds

  1. 1
    kubectl get pods -l app=<name>
    Look for
    STATUS Running with READY 0/1, and RESTARTS not rising.
    What it tells you
    Rules in a readiness problem. If RESTARTS climbs, look at liveness or crashes instead.
  2. 2
    kubectl describe pod <pod>
    Look for
    The Readiness line of the container (port, path, timings), and Unhealthy events with “Readiness probe failed” and the reason after it.
    What it tells you
    A status code means the app answered “not ready”. “connection refused” means nothing listens on that port. “context deadline exceeded” means it answered too slowly.
  3. 3
    kubectl get endpointslices -l kubernetes.io/service-name=<service> -o yaml
    Look for
    Under each endpoint, conditions.ready.
    What it tells you
    ready: false on every endpoint confirms the Service has nowhere to send traffic.
  4. 4
    kubectl exec <pod> -- wget -qO- http://localhost:<port><path>
    Look for
    What the readiness endpoint itself returns from inside the container.
    What it tells you
    Separates a wrong port or path in the probe from an app that really reports not ready, and usually names the failing dependency.
  5. 5
    kubectl logs <pod> --tail=50
    Look for
    Errors about the dependency the readiness handler checks: database, cache, another service.
    What it tells you
    Rules in the root cause behind the “not ready” answer.

Most common first

Root causes

  1. 1A dependency that the readiness endpoint checks is down or slow.

    Confirm
    The probe fails with a 5xx status in every pod at the same moment, and the logs name the dependency.
    Fix now
    Restore the dependency. The pods return by themselves on the next successful check.
    Prevent
    Decide which dependencies really make the pod unable to serve. Serve degraded responses where you can instead of reporting not ready.
  2. 2The probe points at the wrong port, path or scheme.

    Confirm
    The failure reads “connection refused” or a 404, and it began with a config or image change.
    Fix now
    Correct the port, path or scheme in the readinessProbe.
    Prevent
    Name the container port and refer to it by name in the probe.
  3. 3The app answers, but slower than the probe's timeout, which is one second by default.

    Confirm
    Events say “context deadline exceeded” or mention a timeout, mostly under load.
    Fix now
    Raise timeoutSeconds, and make the readiness handler cheap.
    Prevent
    Keep heavy work out of the readiness endpoint.
  4. 4The app is still starting, and the probe gives it too little time.

    Confirm
    New pods stay at 0/1 for a while after each deploy and then recover.
    Fix now
    Wait, or add a startup probe sized for the real boot time.
    Prevent
    Measure start-up time and set the startup probe from it.
  5. 5The pod is overloaded and refuses or queues the probe's connection.

    Confirm
    Pods flip between 1/1 and 0/1 during traffic peaks, and CPU is throttled or at its limit.
    Fix now
    Add replicas or CPU. Pods leaving the Service push more load onto the rest.
    Prevent
    Scale on load before saturation, and give the health endpoint its own small worker pool where the framework allows it.

The tempting wrong fix

Delete or restart the pods

A readiness failure is not a stuck process. The new containers run the same check against the same broken dependency or the same wrong port, and come up not Ready again. You also throw away warm caches and add a start-up wave on top of the outage. Find out what the check is failing on first.

Before / after

The most common fix: point the probe at what the app really serves

Before
containers:
  - name: web
    ports:
      - containerPort: 8080
    readinessProbe:
      httpGet:
        path: /health    # the app serves /ready
        port: 80         # nothing listens on 80
      periodSeconds: 10
After
containers:
  - name: web
    ports:
      - name: http
        containerPort: 8080
    readinessProbe:
      httpGet:
        path: /ready
        port: http       # refers to the named port
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3

Defaults if you leave them out: a check every 10 seconds, a 1 second timeout, 3 failures to become not Ready and 1 success to become Ready again.

In the real world

Depends on version and provider

The probe defaults are upstream and the same everywhere. What differs is how fast traffic follows. kube-proxy, an eBPF dataplane such as Cilium, and cloud load balancers each pick up endpoint changes on their own schedule, and load balancers that send traffic straight to pod IPs (for example with readiness gates on GKE or EKS) run their own health checks as well. What clients see with zero endpoints also varies: typically a refused connection from kube-proxy, and a 503 from an ingress controller or gateway.

Seen in a real outage

The opposite failure is just as real: a check that keeps passing while the server cannot serve. At Val Town in 2024, a fixed group of customers had requests hang for seven hours while the health check stayed green.

Break it: Same customers, same sick server

Your turn

What would you do?

All pods of a Deployment show Running, 0/1 READY and 0 restarts. Clients get 503 from the ingress. Which statement is correct?

Readiness probe failure questions

Why is my pod Running but not Ready?
Its readiness probe is failing. The container is up, but it is telling the kubelet it cannot take requests, so the pod shows 0/1 READY and is left out of the Service until the probe passes.
Does a failed readiness probe restart the pod?
No. A readiness failure only removes the pod from the Service's endpoints. The container keeps running. Restarts come from a failing liveness or startup probe, or from the process exiting.
What happens when all pods fail their readiness probe?
The Service has no ready endpoints. Requests to it fail, typically with a refused connection inside the cluster or a 503 from the ingress controller, until at least one pod passes its probe again.

Sources: Kubernetes docs: Readiness probe ↗ · Kubernetes docs: Configure liveness, readiness and startup probes ↗ · Kubernetes docs: EndpointSlices, conditions ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play