Skip to content
Breaks at kubelet · Your containerCan take the service downCommon

Liveness probe failure

The kubelet decided the container was stuck, killed it and started it again. Often the container was fine and the check was wrong.

What you'll seeIllustrative output
$ kubectl get pods -l app=api
NAME                   READY   STATUS    RESTARTS      AGE
api-5f8d7c9b4-2xk8p    1/1     Running   4 (38s ago)   21m
api-5f8d7c9b4-jw6mn    1/1     Running   4 (41s ago)   21m
api-5f8d7c9b4-r7c2d    0/1     Running   4 (12s ago)   21m
$ kubectl describe pod api-5f8d7c9b4-2xk8p
    Liveness:       http-get http://:8080/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
    Restart Count:  4
Events:
  Type     Reason     Age                  From     Message
  Warning  Unhealthy  42s (x12 over 9m)    kubelet  Liveness probe failed: HTTP probe failed with statuscode: 500
  Normal   Killing    42s (x4 over 9m)     kubelet  Container api failed liveness probe, will be restarted
$ kubectl logs api-5f8d7c9b4-2xk8p --previous --tail=3
GET /healthz 500 1002ms  db ping failed: dial tcp 10.0.4.12:5432: i/o timeout
GET /healthz 500 1001ms  db ping failed: dial tcp 10.0.4.12:5432: i/o timeout
received SIGTERM, shutting down

Watch it happen

How Liveness probe failure unfolds

Step 3 · The failure

The database goes down, so every pod is restarted

The database fails, not the apps. But every health check asks the database, so every check fails. The kubelet restarts all three containers, the new ones fail the same check, and it restarts them again. A database outage has become a full outage of the service.

  • DB timeout / probe failed
  • restart

Where it breaks: The kubelet does the killing, acting on the liveness probe you configured. Either the container really is stuck, or the check fails for a reason a restart cannot fix: a dependency that is down, a timeout that is too short, an app that is still starting.

In short

What Liveness probe failure means

A liveness probe exists for one case: a process that is still running but will never recover by itself, such as a deadlock. The kubelet runs the check on a fixed period. After three failures in a row it kills the container and starts a new one, subject to the pod's restart policy. That is useful when the process is stuck. It is harmful when the check fails for some other reason, because a restart then fixes nothing and throws away a working process. The classic trap is a liveness endpoint that also checks the database. When the database goes down, the check fails in every pod at once, the kubelet restarts them all, the new containers fail the same check, and the pods slide into CrashLoopBackOff. An outage in one dependency has become an outage of the whole service.

In order

Diagnose in 60 seconds

  1. 1
    kubectl get pods -l app=<name>
    Look for
    RESTARTS rising. Note whether it is one pod or all of them, and whether the restarts happen at the same time.
    What it tells you
    One pod: that process is probably stuck or short of resources. All pods together: suspect something they share, such as a dependency behind the check.
  2. 2
    kubectl describe pod <pod>
    Look for
    The Liveness line (endpoint and timings), then Unhealthy events with “Liveness probe failed” and a Killing event saying the container will be restarted.
    What it tells you
    Rules in a liveness kill, as opposed to a crash or an OOM kill. The text after “failed:” is the reason: a status code, a refused connection or a timeout.
  3. 3
    kubectl logs <pod> --previous --tail=50
    Look for
    What the health endpoint logged before the kill, and whether the process was busy, blocked or starting up.
    What it tells you
    Shows whether the app was stuck, or healthy and failing the check because of something else.
  4. 4
    kubectl exec <pod> -- wget -qO- http://localhost:<port><path>
    Look for
    The answer and the time it takes, compared with the probe's timeoutSeconds.
    What it tells you
    Rules in a slow or dependency-bound health endpoint. If it names a database or another service, that is the trap.
  5. 5
    kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[*].livenessProbe}'
    Look for
    timeoutSeconds, periodSeconds, failureThreshold, and whether a startupProbe exists next to it.
    What it tells you
    A one second timeout with no startup probe on a slow-starting or busy app explains kills during boot and under load.

Most common first

Root causes

  1. 1The liveness endpoint checks a dependency (database, cache, another service) that is down or slow.

    Confirm
    Every pod is restarted at about the same time, and the previous logs show the health endpoint failing on the dependency.
    Fix now
    Make the liveness endpoint answer from the process alone. Restore the dependency separately.
    Prevent
    Dependency checks go in readiness, never in liveness.
  2. 2The app is still starting when liveness begins, so it is killed before it has finished booting.

    Confirm
    Restarts happen shortly after each start, and the logs show start-up work in progress when SIGTERM arrives.
    Fix now
    Add a startup probe that allows for the slowest realistic start.
    Prevent
    Do not rely on initialDelaySeconds alone. A startup probe holds liveness back until the app is up.
  3. 3The check times out under load: the default timeout is one second.

    Confirm
    Events mention a timeout or “context deadline exceeded”, and kills cluster around traffic peaks, long garbage-collection pauses or CPU throttling.
    Fix now
    Raise timeoutSeconds and failureThreshold, and remove the CPU limit or raise it if the container is throttled.
    Prevent
    Keep the health handler trivial and off the request worker pool where the framework allows it.
  4. 4The process really is stuck: a deadlock, an exhausted thread pool, an event loop that is blocked.

    Confirm
    Only that pod fails, its logs stop making progress, and it works again after the restart.
    Fix now
    Nothing urgent: the restart was correct. Find the bug that made it hang.
    Prevent
    Alert on liveness restarts so that recurring hangs are investigated, not hidden.
  5. 5The probe points at the wrong port or path.

    Confirm
    The failure reads “connection refused” or a 404 from the very first check after a deploy.
    Fix now
    Correct the probe's port and path.
    Prevent
    Refer to a named container port, and test the probe in a staging namespace.

The tempting wrong fix

Have the liveness probe check the database too

It feels thorough, and it is the trap itself. A restart cannot repair a database, so when the database fails the kubelet kills every healthy pod, repeatedly. You lose warm processes, add a start-up storm against a database that is trying to recover, and turn a partial outage into a total one.

Before / after

The most common fix: liveness for the process, readiness for dependencies

Before
livenessProbe:
  httpGet:
    path: /healthz     # also pings the database
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3
After
startupProbe:           # up to 30 × 10s to boot
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
  failureThreshold: 30
livenessProbe:          # the process only
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3
readinessProbe:         # dependencies go here
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5

Defaults if you leave them out: a check every 10 seconds, a 1 second timeout and 3 failures in a row before the kubelet acts.

In the real world

Depends on version and provider

Probe defaults are the same on every conformant cluster. Two things do vary. After the kill, the container gets the pod's terminationGracePeriodSeconds (30 seconds by default) to stop, or a probe-level value if you set one, so the exit code is typically 143 for a clean stop on SIGTERM and 137 if it had to be killed. And repeated restarts follow the kubelet's crash back-off, whose cap is configurable per node on newer versions, so the gaps between restarts differ between clusters and managed providers.

Seen in a real outage

An automatic repair that makes an outage worse is an old pattern. At Facebook in 2010, a system built to correct a bad configuration value sent every client to the database at once, and kept the site down after the value had been fixed.

Break it: The fix that kept it down

Your turn

What would you do?

During a database outage every pod of the api Deployment is restarted every half minute, with “Liveness probe failed” events. The app processes are healthy. What is the right change?

Liveness probe failure questions

What happens when a liveness probe fails?
After the configured number of failures in a row, three by default, the kubelet kills the container and starts it again according to the pod's restart policy. The pod stays on its node and its restart count goes up.
Should a liveness probe check the database?
No. A restart cannot fix a database, so a database outage would make the kubelet restart every pod at once. Liveness should only report whether the process itself is stuck. Check dependencies in the readiness probe.
Why does my pod restart during start-up?
The liveness probe begins before the application has finished starting, fails three times and the kubelet kills the container. Add a startup probe: liveness and readiness checks are held back until it succeeds.

Sources: Kubernetes docs: Liveness probe ↗ · Kubernetes docs: Configure liveness, readiness and startup probes ↗ · Kubernetes docs: Pod lifecycle, container probes ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play