Skip to content
Breaks at Your container · kubeletCan take the service downVery common

CrashLoopBackOff

The container starts, exits, and is restarted over and over, with a longer wait before each new attempt.

What you'll seeIllustrative output
$ kubectl get pods
NAME                  READY   STATUS             RESTARTS        AGE
api-7d9c5b6f4-x2k8p   0/1     CrashLoopBackOff   6 (2m14s ago)   9m41s
$ kubectl describe pod api-7d9c5b6f4-x2k8p
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
    Restart Count:  6
Events:
  Type     Reason   Age                  From     Message
  Normal   Pulled   2m15s (x7 over 9m)   kubelet  Container image "registry.example.com/api:1.4.2" already present on machine
  Warning  BackOff  4s (x38 over 9m)     kubelet  Back-off restarting failed container api in pod api-7d9c5b6f4-x2k8p_default(…)
$ kubectl logs api-7d9c5b6f4-x2k8p --previous
Error: missing required environment variable DATABASE_URL

Watch it happen

How CrashLoopBackOff unfolds

Step 3 · The failure

Every wait is twice as long

It crashes again each time. The kubelet's wait doubles after every failure: 10s, 20s, 40s, 80s, up to a cap of 300s. Watch the gaps between the violet restarts grow. The container is the problem; the kubelet is only slowing the loop down.

  • exit
  • back-off
  • restart

Where it breaks: Your container is what fails: it exits soon after starting. The kubelet only reports it, and adds the growing wait between restarts.

In short

What CrashLoopBackOff means

CrashLoopBackOff is not an error of its own. It is the kubelet telling you that a container started, exited, and is now waiting to be started again. Scheduling worked and the image was pulled; the process inside the container is what keeps ending. After each exit the kubelet waits before the next attempt, and the wait doubles: 10 seconds, 20, 40, and so on up to five minutes. A container that then runs for ten minutes without trouble gets its wait reset. So the status is a symptom, and the cause is whatever made the process exit: a missing environment variable, a dependency it cannot reach, a wrong command, an out-of-memory kill, or a liveness probe that kills it before it has finished starting. The exit code and the logs of the previous attempt tell you which.

In order

Diagnose in 60 seconds

  1. 1
    kubectl get pod <pod>
    Look for
    STATUS CrashLoopBackOff and a RESTARTS count that keeps rising.
    What it tells you
    Rules in: the container starts and then exits. Rules out: scheduling and image pulls, which fail before a container ever runs.
  2. 2
    kubectl describe pod <pod>
    Look for
    Under Last State: the Reason and the Exit Code. Then the Events at the bottom.
    What it tells you
    Exit 1 or 2: the app failed. 137 with Reason OOMKilled: memory. 137 with “Liveness probe failed” events: the kubelet killed it. 126 or 127: the command cannot be run or found. 0 with Completed: the process simply finished.
  3. 3
    kubectl logs <pod> --previous
    Look for
    The last lines printed by the attempt that crashed. Without --previous you get the new, often empty, container.
    What it tells you
    Usually names the cause directly: a missing variable, a refused connection, a stack trace.
  4. 4
    kubectl get events --field-selector involvedObject.name=<pod>
    Look for
    Unhealthy, Killing, FailedMount or config errors around the time of each restart.
    What it tells you
    Rules in a probe or volume problem when the logs look clean.
  5. 5
    kubectl rollout history deployment/<name>
    Look for
    Whether the crashes began with the latest revision.
    What it tells you
    Rules in a bad release. kubectl rollout undo returns to the previous one while you investigate.

Most common first

Root causes

  1. 1The app fails on startup: a missing or wrong environment variable, Secret or config file.

    Confirm
    The previous logs name the missing setting, and the exit code is 1.
    Fix now
    Add or correct the value in the Deployment, ConfigMap or Secret and let the pods roll.
    Prevent
    Validate configuration at startup with a clear message, and deploy to a staging namespace first.
  2. 2A dependency is unreachable at startup (database, queue, another service) and the app exits instead of retrying.

    Confirm
    Logs end with a connection refused, a timeout or a DNS error.
    Fix now
    Restore the dependency or the network path to it.
    Prevent
    Retry connections inside the app with backoff. Report “not ready” through a readiness probe instead of exiting.
  3. 3The container is killed for memory: OOMKilled.

    Confirm
    Last State shows Reason OOMKilled and Exit Code 137.
    Fix now
    See the OOMKilled page: find out whether usage levels off, then set the limit to fit.
    Prevent
    Set memory requests and limits from measured usage.
  4. 4A liveness probe kills the container before it has finished starting.

    Confirm
    Events show “Liveness probe failed” followed by “Killing”, and the exit code is 137 or 143.
    Fix now
    Add a startup probe that allows for the real start-up time.
    Prevent
    Never rely on initialDelaySeconds alone for slow starters.
  5. 5The command, arguments or image are wrong: a typo in the entrypoint, or an image built for another CPU architecture.

    Confirm
    Exit code 126 or 127, or “exec format error” in the logs.
    Fix now
    Correct command and args, or build and pull the image for the node's architecture.
    Prevent
    Run the image locally with the same command before deploying it.
  6. 6The process finishes normally (exit 0) but runs under a controller that expects it to stay up.

    Confirm
    Last State shows Reason Completed and Exit Code 0.
    Fix now
    Run one-off work as a Job, or keep the main process in the foreground.
    Prevent
    Match the workload type to the lifetime of the process.

The tempting wrong fix

Delete the pod

The Deployment's ReplicaSet creates a replacement from the same template, and it crashes for the same reason. All you gain is a reset back-off timer, and you lose the previous container's logs, which were your best evidence.

Before / after

The most common fix: give the container the setting it is missing

Before
containers:
  - name: api
    image: registry.example.com/api:1.4.2
    # the app reads DATABASE_URL at startup
    # and exits with code 1 when it is not set
After
containers:
  - name: api
    image: registry.example.com/api:1.4.2
    env:
      - name: DATABASE_URL
        valueFrom:
          secretKeyRef:
            name: api-secrets
            key: database-url

If the Secret or the key does not exist, the pod will not crash-loop: it stays in CreateContainerConfigError, which is a different problem with a different fix.

In the real world

Depends on version and provider

The 10 second start, the doubling and the 300 second cap are upstream defaults. The cap can be lowered per node through the kubelet's crashLoopBackOff.maxContainerRestartPeriod setting, which is on by default from Kubernetes v1.35, and an alpha feature gate (ReduceDefaultCrashLoopBackOffDecay) starts at 1 second with a 60 second cap. Managed providers may set either, so the gaps you observe can be shorter.

Seen in a real outage

A crash loop often has a trigger that survives the restart. At incident.io in 2022, one bad message crashed the app that read it, and every restart picked the same message up again.

Break it: The message that kills its reader

Your turn

What would you do?

A pod shows CrashLoopBackOff with 12 restarts. kubectl logs prints nothing. What should you run next to see why the container died?

CrashLoopBackOff questions

What does CrashLoopBackOff mean in Kubernetes?
A container in the pod keeps exiting shortly after it starts. The kubelet restarts it each time, waiting longer before every attempt. CrashLoopBackOff is the status shown during that wait.
How long is the CrashLoopBackOff delay?
By default it starts at 10 seconds and doubles after each failed restart, up to 300 seconds. It resets once the container has run for 10 minutes without a problem. Both limits can be configured on newer clusters.
How do I fix CrashLoopBackOff?
Find out why the process exits. Read the exit code with kubectl describe pod and the output of the crashed attempt with kubectl logs --previous, then fix that cause: configuration, a dependency, memory, a probe or the command.

Sources: Kubernetes docs: Pod lifecycle, container restart policy ↗ · Kubernetes docs: Debug running pods ↗ · Kubernetes docs: Determine the reason for pod failure ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play