Liveness probe failure
The kubelet decided the container was stuck, killed it and started it again. Often the container was fine and the check was wrong.
$ kubectl get pods -l app=apiNAME READY STATUS RESTARTS AGE api-5f8d7c9b4-2xk8p 1/1 Running 4 (38s ago) 21m api-5f8d7c9b4-jw6mn 1/1 Running 4 (41s ago) 21m api-5f8d7c9b4-r7c2d 0/1 Running 4 (12s ago) 21m
$ kubectl describe pod api-5f8d7c9b4-2xk8p Liveness: http-get http://:8080/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Last State: Terminated
Reason: Error
Exit Code: 137
Restart Count: 4
Events:
Type Reason Age From Message
Warning Unhealthy 42s (x12 over 9m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 500
Normal Killing 42s (x4 over 9m) kubelet Container api failed liveness probe, will be restarted$ kubectl logs api-5f8d7c9b4-2xk8p --previous --tail=3GET /healthz 500 1002ms db ping failed: dial tcp 10.0.4.12:5432: i/o timeout GET /healthz 500 1001ms db ping failed: dial tcp 10.0.4.12:5432: i/o timeout received SIGTERM, shutting down
Watch it happen
How Liveness probe failure unfolds
Step 3 · The failure
The database goes down, so every pod is restarted
The database fails, not the apps. But every health check asks the database, so every check fails. The kubelet restarts all three containers, the new ones fail the same check, and it restarts them again. A database outage has become a full outage of the service.
- DB timeout / probe failed
- restart
Step 3 · The failure
The database goes down, so every pod is restarted
The database fails, not the apps. But every health check asks the database, so every check fails. The kubelet restarts all three containers, the new ones fail the same check, and it restarts them again. A database outage has become a full outage of the service.
- DB timeout / probe failed
- restart
Where it breaks: The kubelet does the killing, acting on the liveness probe you configured. Either the container really is stuck, or the check fails for a reason a restart cannot fix: a dependency that is down, a timeout that is too short, an app that is still starting.
In short
What Liveness probe failure means
A liveness probe exists for one case: a process that is still running but will never recover by itself, such as a deadlock. The kubelet runs the check on a fixed period. After three failures in a row it kills the container and starts a new one, subject to the pod's restart policy. That is useful when the process is stuck. It is harmful when the check fails for some other reason, because a restart then fixes nothing and throws away a working process. The classic trap is a liveness endpoint that also checks the database. When the database goes down, the check fails in every pod at once, the kubelet restarts them all, the new containers fail the same check, and the pods slide into CrashLoopBackOff. An outage in one dependency has become an outage of the whole service.
In order
Diagnose in 60 seconds
- 1
kubectl get pods -l app=<name>- Look for
- RESTARTS rising. Note whether it is one pod or all of them, and whether the restarts happen at the same time.
- What it tells you
- One pod: that process is probably stuck or short of resources. All pods together: suspect something they share, such as a dependency behind the check.
- 2
kubectl describe pod <pod>- Look for
- The Liveness line (endpoint and timings), then Unhealthy events with “Liveness probe failed” and a Killing event saying the container will be restarted.
- What it tells you
- Rules in a liveness kill, as opposed to a crash or an OOM kill. The text after “failed:” is the reason: a status code, a refused connection or a timeout.
- 3
kubectl logs <pod> --previous --tail=50- Look for
- What the health endpoint logged before the kill, and whether the process was busy, blocked or starting up.
- What it tells you
- Shows whether the app was stuck, or healthy and failing the check because of something else.
- 4
kubectl exec <pod> -- wget -qO- http://localhost:<port><path>- Look for
- The answer and the time it takes, compared with the probe's timeoutSeconds.
- What it tells you
- Rules in a slow or dependency-bound health endpoint. If it names a database or another service, that is the trap.
- 5
kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[*].livenessProbe}'- Look for
- timeoutSeconds, periodSeconds, failureThreshold, and whether a startupProbe exists next to it.
- What it tells you
- A one second timeout with no startup probe on a slow-starting or busy app explains kills during boot and under load.
Most common first
Root causes
1The liveness endpoint checks a dependency (database, cache, another service) that is down or slow.
- Confirm
- Every pod is restarted at about the same time, and the previous logs show the health endpoint failing on the dependency.
- Fix now
- Make the liveness endpoint answer from the process alone. Restore the dependency separately.
- Prevent
- Dependency checks go in readiness, never in liveness.
2The app is still starting when liveness begins, so it is killed before it has finished booting.
- Confirm
- Restarts happen shortly after each start, and the logs show start-up work in progress when SIGTERM arrives.
- Fix now
- Add a startup probe that allows for the slowest realistic start.
- Prevent
- Do not rely on initialDelaySeconds alone. A startup probe holds liveness back until the app is up.
3The check times out under load: the default timeout is one second.
- Confirm
- Events mention a timeout or “context deadline exceeded”, and kills cluster around traffic peaks, long garbage-collection pauses or CPU throttling.
- Fix now
- Raise timeoutSeconds and failureThreshold, and remove the CPU limit or raise it if the container is throttled.
- Prevent
- Keep the health handler trivial and off the request worker pool where the framework allows it.
4The process really is stuck: a deadlock, an exhausted thread pool, an event loop that is blocked.
- Confirm
- Only that pod fails, its logs stop making progress, and it works again after the restart.
- Fix now
- Nothing urgent: the restart was correct. Find the bug that made it hang.
- Prevent
- Alert on liveness restarts so that recurring hangs are investigated, not hidden.
5The probe points at the wrong port or path.
- Confirm
- The failure reads “connection refused” or a 404 from the very first check after a deploy.
- Fix now
- Correct the probe's port and path.
- Prevent
- Refer to a named container port, and test the probe in a staging namespace.
The tempting wrong fix
Have the liveness probe check the database too
Before / after
The most common fix: liveness for the process, readiness for dependencies
livenessProbe:
httpGet:
path: /healthz # also pings the database
port: 8080
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3startupProbe: # up to 30 × 10s to boot
httpGet: { path: /livez, port: 8080 }
periodSeconds: 10
failureThreshold: 30
livenessProbe: # the process only
httpGet: { path: /livez, port: 8080 }
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
readinessProbe: # dependencies go here
httpGet: { path: /readyz, port: 8080 }
periodSeconds: 5Defaults if you leave them out: a check every 10 seconds, a 1 second timeout and 3 failures in a row before the kubelet acts.
In the real world
Depends on version and provider
Seen in a real outage
An automatic repair that makes an outage worse is an old pattern. At Facebook in 2010, a system built to correct a bad configuration value sent every client to the database at once, and kept the site down after the value had been fixed.
Break it: The fix that kept it downYour turn
What would you do?
During a database outage every pod of the api Deployment is restarted every half minute, with “Liveness probe failed” events. The app processes are healthy. What is the right change?
Related errors
Liveness probe failure questions
- What happens when a liveness probe fails?
- After the configured number of failures in a row, three by default, the kubelet kills the container and starts it again according to the pod's restart policy. The pod stays on its node and its restart count goes up.
- Should a liveness probe check the database?
- No. A restart cannot fix a database, so a database outage would make the kubelet restart every pod at once. Liveness should only report whether the process itself is stuck. Check dependencies in the readiness probe.
- Why does my pod restart during start-up?
- The liveness probe begins before the application has finished starting, fails three times and the kubelet kills the container. Add a startup probe: liveness and readiness checks are held back until it succeeds.
Sources: Kubernetes docs: Liveness probe ↗ · Kubernetes docs: Configure liveness, readiness and startup probes ↗ · Kubernetes docs: Pod lifecycle, container probes ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.