CrashLoopBackOff
The container starts, exits, and is restarted over and over, with a longer wait before each new attempt.
$ kubectl get podsNAME READY STATUS RESTARTS AGE api-7d9c5b6f4-x2k8p 0/1 CrashLoopBackOff 6 (2m14s ago) 9m41s
$ kubectl describe pod api-7d9c5b6f4-x2k8p State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Restart Count: 6
Events:
Type Reason Age From Message
Normal Pulled 2m15s (x7 over 9m) kubelet Container image "registry.example.com/api:1.4.2" already present on machine
Warning BackOff 4s (x38 over 9m) kubelet Back-off restarting failed container api in pod api-7d9c5b6f4-x2k8p_default(…)$ kubectl logs api-7d9c5b6f4-x2k8p --previousError: missing required environment variable DATABASE_URL
Watch it happen
How CrashLoopBackOff unfolds
Step 3 · The failure
Every wait is twice as long
It crashes again each time. The kubelet's wait doubles after every failure: 10s, 20s, 40s, 80s, up to a cap of 300s. Watch the gaps between the violet restarts grow. The container is the problem; the kubelet is only slowing the loop down.
- exit
- back-off
- restart
Step 3 · The failure
Every wait is twice as long
It crashes again each time. The kubelet's wait doubles after every failure: 10s, 20s, 40s, 80s, up to a cap of 300s. Watch the gaps between the violet restarts grow. The container is the problem; the kubelet is only slowing the loop down.
- exit
- back-off
- restart
Where it breaks: Your container is what fails: it exits soon after starting. The kubelet only reports it, and adds the growing wait between restarts.
In short
What CrashLoopBackOff means
CrashLoopBackOff is not an error of its own. It is the kubelet telling you that a container started, exited, and is now waiting to be started again. Scheduling worked and the image was pulled; the process inside the container is what keeps ending. After each exit the kubelet waits before the next attempt, and the wait doubles: 10 seconds, 20, 40, and so on up to five minutes. A container that then runs for ten minutes without trouble gets its wait reset. So the status is a symptom, and the cause is whatever made the process exit: a missing environment variable, a dependency it cannot reach, a wrong command, an out-of-memory kill, or a liveness probe that kills it before it has finished starting. The exit code and the logs of the previous attempt tell you which.
In order
Diagnose in 60 seconds
- 1
kubectl get pod <pod>- Look for
- STATUS CrashLoopBackOff and a RESTARTS count that keeps rising.
- What it tells you
- Rules in: the container starts and then exits. Rules out: scheduling and image pulls, which fail before a container ever runs.
- 2
kubectl describe pod <pod>- Look for
- Under Last State: the Reason and the Exit Code. Then the Events at the bottom.
- What it tells you
- Exit 1 or 2: the app failed. 137 with Reason OOMKilled: memory. 137 with “Liveness probe failed” events: the kubelet killed it. 126 or 127: the command cannot be run or found. 0 with Completed: the process simply finished.
- 3
kubectl logs <pod> --previous- Look for
- The last lines printed by the attempt that crashed. Without --previous you get the new, often empty, container.
- What it tells you
- Usually names the cause directly: a missing variable, a refused connection, a stack trace.
- 4
kubectl get events --field-selector involvedObject.name=<pod>- Look for
- Unhealthy, Killing, FailedMount or config errors around the time of each restart.
- What it tells you
- Rules in a probe or volume problem when the logs look clean.
- 5
kubectl rollout history deployment/<name>- Look for
- Whether the crashes began with the latest revision.
- What it tells you
- Rules in a bad release. kubectl rollout undo returns to the previous one while you investigate.
Most common first
Root causes
1The app fails on startup: a missing or wrong environment variable, Secret or config file.
- Confirm
- The previous logs name the missing setting, and the exit code is 1.
- Fix now
- Add or correct the value in the Deployment, ConfigMap or Secret and let the pods roll.
- Prevent
- Validate configuration at startup with a clear message, and deploy to a staging namespace first.
2A dependency is unreachable at startup (database, queue, another service) and the app exits instead of retrying.
- Confirm
- Logs end with a connection refused, a timeout or a DNS error.
- Fix now
- Restore the dependency or the network path to it.
- Prevent
- Retry connections inside the app with backoff. Report “not ready” through a readiness probe instead of exiting.
3The container is killed for memory: OOMKilled.
- Confirm
- Last State shows Reason OOMKilled and Exit Code 137.
- Fix now
- See the OOMKilled page: find out whether usage levels off, then set the limit to fit.
- Prevent
- Set memory requests and limits from measured usage.
4A liveness probe kills the container before it has finished starting.
- Confirm
- Events show “Liveness probe failed” followed by “Killing”, and the exit code is 137 or 143.
- Fix now
- Add a startup probe that allows for the real start-up time.
- Prevent
- Never rely on initialDelaySeconds alone for slow starters.
5The command, arguments or image are wrong: a typo in the entrypoint, or an image built for another CPU architecture.
- Confirm
- Exit code 126 or 127, or “exec format error” in the logs.
- Fix now
- Correct command and args, or build and pull the image for the node's architecture.
- Prevent
- Run the image locally with the same command before deploying it.
6The process finishes normally (exit 0) but runs under a controller that expects it to stay up.
- Confirm
- Last State shows Reason Completed and Exit Code 0.
- Fix now
- Run one-off work as a Job, or keep the main process in the foreground.
- Prevent
- Match the workload type to the lifetime of the process.
The tempting wrong fix
Delete the pod
Before / after
The most common fix: give the container the setting it is missing
containers:
- name: api
image: registry.example.com/api:1.4.2
# the app reads DATABASE_URL at startup
# and exits with code 1 when it is not setcontainers:
- name: api
image: registry.example.com/api:1.4.2
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: api-secrets
key: database-urlIf the Secret or the key does not exist, the pod will not crash-loop: it stays in CreateContainerConfigError, which is a different problem with a different fix.
In the real world
Depends on version and provider
Seen in a real outage
A crash loop often has a trigger that survives the restart. At incident.io in 2022, one bad message crashed the app that read it, and every restart picked the same message up again.
Break it: The message that kills its readerYour turn
What would you do?
A pod shows CrashLoopBackOff with 12 restarts. kubectl logs prints nothing. What should you run next to see why the container died?
Related errors
CrashLoopBackOff questions
- What does CrashLoopBackOff mean in Kubernetes?
- A container in the pod keeps exiting shortly after it starts. The kubelet restarts it each time, waiting longer before every attempt. CrashLoopBackOff is the status shown during that wait.
- How long is the CrashLoopBackOff delay?
- By default it starts at 10 seconds and doubles after each failed restart, up to 300 seconds. It resets once the container has run for 10 minutes without a problem. Both limits can be configured on newer clusters.
- How do I fix CrashLoopBackOff?
- Find out why the process exits. Read the exit code with kubectl describe pod and the output of the crashed attempt with kubectl logs --previous, then fix that cause: configuration, a dependency, memory, a probe or the command.
Sources: Kubernetes docs: Pod lifecycle, container restart policy ↗ · Kubernetes docs: Debug running pods ↗ · Kubernetes docs: Determine the reason for pod failure ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.