Skip to content
Breaks at Image registry · kubeletDegrades the serviceVery common

ImagePullBackOff

The node cannot download the container image, so the container never starts. The kubelet keeps retrying, more and more slowly.

What you'll seeIllustrative output
$ kubectl get pods
NAME                  READY   STATUS             RESTARTS   AGE
api-6f8d4c7b59-t7xvw   0/1     ImagePullBackOff   0          3m12s
$ kubectl describe pod api-6f8d4c7b59-t7xvw
    State:          Waiting
      Reason:       ImagePullBackOff
Events:
  Type     Reason     Age                  From               Message
  Normal   Scheduled  3m12s                default-scheduler  Successfully assigned default/api-6f8d4c7b59-t7xvw to node-a
  Normal   Pulling    98s (x4 over 3m11s)  kubelet            Pulling image "registry.example.com/api:1.4.3"
  Warning  Failed     97s (x4 over 3m10s)  kubelet            Failed to pull image "registry.example.com/api:1.4.3": rpc error: code = NotFound desc = failed to pull and unpack image "registry.example.com/api:1.4.3": failed to resolve reference "registry.example.com/api:1.4.3": registry.example.com/api:1.4.3: not found
  Warning  Failed     97s (x4 over 3m10s)  kubelet            Error: ErrImagePull
  Normal   BackOff    20s (x6 over 3m9s)   kubelet            Back-off pulling image "registry.example.com/api:1.4.3"
  Warning  Failed     20s (x6 over 3m9s)   kubelet            Error: ImagePullBackOff

Watch it happen

How ImagePullBackOff unfolds

Step 3 · The failure

Same request, same refusal, longer waits

Every retry asks for the same missing tag and gets the same answer. The kubelet's wait grows after each failure, up to a cap of 300 seconds. The container never starts, so RESTARTS stays at 0 and there are no logs.

  • retry
  • refused
  • back-off

Where it breaks: The registry is where the request is refused, usually because of what the pod asked for: a name or tag that does not exist, or no credentials. The kubelet reports ErrImagePull and then ImagePullBackOff while it waits to try again.

In short

What ImagePullBackOff means

ImagePullBackOff means the container has never run, because its image could not be downloaded. The kubelet asks the container runtime for the image, the runtime asks the registry, and the registry says no. The first failure is reported as ErrImagePull. After that the kubelet waits before each new attempt, with a delay that grows up to five minutes, and the pod shows ImagePullBackOff during the wait. The pod has been scheduled and the node is fine. Nothing inside your application is involved, so there are no container logs to read. The registry's own answer is in the pod's events, and it nearly always names the cause: the image or tag does not exist, the registry wants credentials the pod does not have, the node has hit a pull rate limit, or the image was not built for the node's CPU architecture.

In order

Diagnose in 60 seconds

  1. 1
    kubectl get pod <pod>
    Look for
    STATUS ImagePullBackOff or ErrImagePull, with RESTARTS at 0.
    What it tells you
    Rules in: the image never arrived. Rules out: scheduling, which already worked, and your application, which has not run yet.
  2. 2
    kubectl describe pod <pod>
    Look for
    The “Failed to pull image” event. Read the end of its message: it is the registry's own answer.
    What it tells you
    “not found” or “manifest unknown”: wrong name or tag. “unauthorized” or “pull access denied”: credentials. “toomanyrequests”: rate limit. “no match for platform”: wrong CPU architecture. A timeout or “no such host”: the node cannot reach the registry.
  3. 3
    kubectl get pod <pod> -o jsonpath='{.spec.containers[*].image}{"\n"}'
    Look for
    The exact image reference the pod asks for: registry host, repository, tag or digest.
    What it tells you
    Rules in a typo, a tag that was never pushed, or a missing registry host that makes the node look on Docker Hub.
  4. 4
    kubectl get pod <pod> -o jsonpath='{.spec.imagePullSecrets}{"\n"}'
    Look for
    The names of the pull secrets attached to the pod. An empty result means it pulls anonymously.
    What it tells you
    Rules in missing credentials for a private registry.
  5. 5
    kubectl get secret <pull-secret> -n <namespace>
    Look for
    That the secret exists in the pod's own namespace and has type kubernetes.io/dockerconfigjson.
    What it tells you
    Rules in a secret that is missing, in another namespace, or of the wrong type. If it exists, its stored password or token may have expired.

Most common first

Root causes

  1. 1The image name or tag is wrong, or the tag was never pushed.

    Confirm
    The event ends in “not found” or “manifest unknown”.
    Fix now
    Correct the image reference in the Deployment, or push the missing tag.
    Prevent
    Have the pipeline deploy the exact tag or digest it just pushed, instead of typing it by hand.
  2. 2The registry is private and the pod has no valid credentials.

    Confirm
    The event says “unauthorized”, “authentication required” or “pull access denied”.
    Fix now
    Create a docker-registry Secret in the pod's namespace and name it under imagePullSecrets, or attach it to the pod's ServiceAccount.
    Prevent
    Attach the pull secret to the namespace's ServiceAccount so new workloads get it by default, and rotate tokens before they expire.
  3. 3The registry is rate-limiting the node's pulls.

    Confirm
    The event contains “toomanyrequests” or status 429.
    Fix now
    Wait for the limit to reset, or pull with an authenticated account that has a higher allowance.
    Prevent
    Mirror public images into your own registry, and avoid imagePullPolicy: Always where a fixed tag would do.
  4. 4The image has no build for the node's CPU architecture.

    Confirm
    The event says “no match for platform in manifest”. The node is arm64 and the image is amd64 only, or the reverse.
    Fix now
    Build and push a multi-architecture image, or schedule the pod onto nodes of the matching architecture.
    Prevent
    Build for every architecture your node pools use.
  5. 5The node cannot reach the registry: DNS, a firewall, a proxy or an untrusted certificate.

    Confirm
    The event shows a timeout, “no such host” or an x509 certificate error instead of an answer from the registry.
    Fix now
    Restore the network path from the node to the registry, or install the registry's CA certificate on the nodes.
    Prevent
    Test a pull from a node whenever network rules or registry certificates change.

The tempting wrong fix

Delete the pod and let it try again

The kubelet is already retrying by itself. A replacement pod asks for the same image with the same credentials and gets the same refusal. Deleting it only resets the back-off, which sends requests faster at a registry that may be rate-limiting you.

Before / after

A common fix: give the pod credentials for a private registry

Before
spec:
  containers:
    - name: api
      image: registry.example.com/api:1.4.2
  # no imagePullSecrets: the node pulls anonymously
  # and the private registry answers “unauthorized”
After
spec:
  imagePullSecrets:
    - name: registry-credentials   # same namespace as the pod
  containers:
    - name: api
      image: registry.example.com/api:1.4.2

Create the Secret first with kubectl create secret docker-registry. If the event says “not found” instead of “unauthorized”, credentials are not the problem: fix the image name or tag.

In the real world

Depends on version and provider

The 300 second cap on the back-off is compiled into the kubelet. On managed clusters, nodes can often pull from the provider's own registry without imagePullSecrets, through the node's cloud identity (for example ECR on EKS, Artifact Registry on GKE, ACR on AKS), so the same manifest can work on one cluster and fail on another. Docker Hub limits anonymous pulls per IP address, and nodes behind one NAT gateway share that allowance. An image built for the wrong architecture that is still pulled successfully fails later, as CrashLoopBackOff with “exec format error”.

Your turn

What would you do?

A pod is in ImagePullBackOff. Its events say: Failed to pull image “registry.example.com/api:1.4.2”: … pull access denied, repository does not exist or may require authorization. What should you check first?

ImagePullBackOff questions

What does ImagePullBackOff mean in Kubernetes?
The kubelet could not pull the container image, so the container cannot start. It keeps trying, waiting longer between attempts, and shows ImagePullBackOff during each wait.
What is the difference between ErrImagePull and ImagePullBackOff?
They are two moments of the same problem. ErrImagePull is reported when a pull attempt has just failed. ImagePullBackOff is reported while the kubelet waits before the next attempt.
How do I fix ImagePullBackOff?
Run kubectl describe pod and read the “Failed to pull image” event. Then fix what it names: the image name or tag, the pull credentials, a registry rate limit, the image's architecture, or the node's network path to the registry.

Sources: Kubernetes docs: Images, ImagePullBackOff ↗ · Kubernetes docs: Images, image pull policy ↗ · Kubernetes docs: Pull an image from a private registry ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play