Skip to content

What's the difference?

Readiness vs liveness vs startup probes

Three pods behind one Service, each with a different probe. The same thing goes wrong in all three at the same moment. Watch what the kubelet does to each.

Scenario 1 · Hang

All three apps hang

At the same moment every app stops answering. Readiness: after three failed checks the pod is pulled from the Service and gets no traffic, but nobody restarts it. Liveness: the kubelet restarts the container and it recovers, after serving errors in between. Startup: nothing happens. That probe finished long ago, so the pod keeps receiving requests it cannot answer.

  • probe / request
  • failed
  • not yet / NotReady
  • restart
  • ok
What you'll see · HangIllustrative output
$ kubectl get pods
NAME            READY   STATUS    RESTARTS      AGE
readiness-pod   0/1     Running   0             12m
liveness-pod    1/1     Running   1 (20s ago)   12m
startup-pod     1/1     Running   0             12m
$ kubectl describe pod readiness-pod
Events:
  Type     Reason     Age               From     Message
  Warning  Unhealthy  8s (x6 over 58s)  kubelet  Readiness probe failed: Get "http://10.244.1.21:8080/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
$ kubectl describe pod liveness-pod
Events:
  Type     Reason     Age                From     Message
  Warning  Unhealthy  50s (x3 over 70s)  kubelet  Liveness probe failed: Get "http://10.244.1.22:8080/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Killing    50s                kubelet  Container api failed liveness probe, will be restarted
  1. ReadinessReadiness: the pod is pulled out of the Service. Traffic stops, the container is not restarted.
  2. LivenessLiveness: the kubelet restarts the container. It heals from a hang, and it kills a slow starter.
  3. StartupStartup: only matters while booting. It does nothing about a later hang.

How to tell them apart

The question it asks

Readiness
Can you take requests right now?
Liveness
Are you stuck beyond recovery?
Startup
Have you finished starting?

When it fails

Readiness
Pod removed from the Service's endpoints
Liveness
Container killed, then restarted
Startup
Container killed, then restarted

Restarts the container?

Readiness
No, never
Liveness
Yes
Startup
Yes, but only during start-up

Stops traffic?

Readiness
Yes, until it passes again
Liveness
No. A restart does that only as a side effect
Startup
Yes: the pod is not ready until it has started

When it runs

Readiness
For the whole life of the container
Liveness
For the whole life of the container
Startup
Only until its first success

Without one

Readiness
Ready as soon as the container has started
Liveness
Never restarted for being stuck
Startup
Liveness and readiness begin immediately

Defaults

Readiness
Every 10s, 1s timeout, 3 failures
Liveness
Every 10s, 1s timeout, 3 failures
Startup
Every 10s, 1s timeout, 3 failures

In short

Readiness vs liveness vs startup probes: the difference

Readiness decides whether a pod gets traffic. Liveness decides whether its container is restarted. Startup holds the other two back until the app has started.

The three probes ask different questions and trigger different actions. A readiness probe asks whether the container can take requests right now. When it fails, the pod is taken out of the Service's endpoints and stops receiving traffic; the container is left alone and returns when the probe passes again. A liveness probe asks whether the container is beyond recovery. When it fails, the kubelet kills the container and restarts it; traffic is not touched until then. A startup probe asks whether the application has finished starting. Until it succeeds, the other two probes are not run, which protects slow starters from being killed early; after that it never runs again. All three share the same defaults: a check every 10 seconds, a one second timeout, and three failures in a row before acting. Most services want readiness and liveness together, plus startup when boot is slow.

All three together

YAML
containers:
  - name: api
    startupProbe:          # about 5 min to boot (30 checks, 10s apart)
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
      failureThreshold: 30
    livenessProbe:         # is the process itself stuck?
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
      failureThreshold: 3
    readinessProbe:        # can it serve right now?
      httpGet: { path: /ready, port: 8080 }
      periodSeconds: 5
      failureThreshold: 3

Keep the liveness endpoint about the process only. If it checks the database, a database outage makes the kubelet restart every pod at once. Put dependency checks in readiness instead.

Interactive

Solve it

Case A

The Spring Boot app that never comes up

A Spring Boot service needs about a minute to start in production. Since a liveness probe was added last week it never becomes Ready, and its restart count keeps climbing.

kubectl outputIllustrative output
$ kubectl get pods
NAME                      READY   STATUS    RESTARTS      AGE
orders-5d9c7b6f48-t7xkp   0/1     Running   4 (25s ago)   3m10s
$ kubectl describe pod orders-5d9c7b6f48-t7xkp
    Liveness:  http-get http://:8080/actuator/health/liveness delay=0s timeout=1s period=10s #success=1 #failure=3
Events:
  Warning  Unhealthy  25s (x12 over 2m55s)  kubelet  Liveness probe failed: Get "http://10.244.2.9:8080/actuator/health/liveness": dial tcp 10.244.2.9:8080: connect: connection refused
  Normal   Killing    25s (x4 over 2m35s)   kubelet  Container orders failed liveness probe, will be restarted

1 · What is wrong?

Try the numbers

How much time does a probe give you?

Move the sliders to see how much time a probe gives your app before something happens. The first check runs after initialDelaySeconds (straight after the container starts when that is 0), then one every periodSeconds. The action comes at the failureThreshold-th failure in a row, so at about initialDelaySeconds + (failureThreshold − 1) × periodSeconds.

budget about 20s
app up 45skilled about 20s
● failed check● passed check■ budget = initialDelay + (failureThreshold − 1) × period

Too slow. Killed at about 20s (plus up to the 1s timeout). The app needs 45s, so it never finishes starting, and the same happens after every restart.

Once running, an app that hangs is restarted after about 21–31s: 3 failed checks 10s apart, each waiting up to the 1s timeout.

Tap to answer

Which probe do I need?

1

Can the app take longer to start than your liveness probe would wait, about (failureThreshold − 1) × periodSeconds?

2

Can the process get stuck in a state that only a restart fixes, such as a deadlock, while it keeps running?

3

Does the pod receive traffic through a Service?

How a probe checks

httpGet
Passes when: The response status is 200 or higher and below 400. The usual choice for web services. Redirects are followed.
tcpSocket
Passes when: The port accepts a connection. Only proves something is listening, not that the app works.
exec
Passes when: The command exits with status 0. Runs inside the container on every check, so keep it cheap.
grpc
Passes when: The gRPC health service answers SERVING. The app must implement the standard gRPC health checking protocol.

All three probe types can use any mechanism. successThreshold defaults to 1 and must be 1 for liveness and startup probes; only readiness can require several passes in a row before a pod counts as ready again.

Framework note

Spring Boot Actuator

Spring Boot's Actuator provides separate liveness and readiness health groups at /actuator/health/liveness and /actuator/health/readiness. They are enabled automatically when the app detects it runs on Kubernetes, and can be switched on or off with management.endpoint.health.probes.enabled. If Actuator runs on a separate management port, a probe there can pass while the main port is stuck, so Spring Boot can also expose the groups on the main port as /livez and /readyz. Spring's own guidance matches this page: the liveness group should not depend on external systems, and adding one to readiness is a judgement call.

Properties
# application.properties
management.endpoint.health.probes.enabled=true
# also serve them on the main port as /livez and /readyz
management.endpoint.health.probes.add-additional-paths=true
# add your own check to readiness (never to liveness)
management.endpoint.health.group.readiness.include=readinessState,customCheck

Source: Spring Boot reference: Actuator endpoints, Kubernetes probes ↗ · Report a problem

Your turn

Quiz: 3 questions

Which probe is this? During a database outage, every pod of a service is restarted again and again, although the app processes themselves are healthy.

A pod has shown Running, 0/1 READY and 0 restarts for ten minutes. Which probe is failing?

An app needs 45 seconds to start. Its only probe is liveness with periodSeconds 10, failureThreshold 3 and no initial delay. What happens?

Questions

What is the difference between a readiness probe and a liveness probe?
A failing readiness probe removes the pod from the Service's endpoints, so it stops receiving traffic, and leaves the container running. A failing liveness probe makes the kubelet kill and restart the container.
When do I need a startup probe?
When the application can take longer to start than the liveness probe would tolerate. The startup probe holds liveness and readiness checks back until the app is up, so a slow start is not mistaken for a hang.
Does a readiness probe restart the pod?
No. A readiness failure only stops traffic to the pod. The container keeps running and starts receiving traffic again as soon as the probe passes.

Sources: Kubernetes docs: Liveness, readiness and startup probes ↗ · Kubernetes docs: Configure liveness, readiness and startup probes ↗ · Kubernetes docs: Pod lifecycle, container probes ↗ · Report a problem

Diagrams are our own simulations. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play