ImagePullBackOff
The node cannot download the container image, so the container never starts. The kubelet keeps retrying, more and more slowly.
$ kubectl get podsNAME READY STATUS RESTARTS AGE api-6f8d4c7b59-t7xvw 0/1 ImagePullBackOff 0 3m12s
$ kubectl describe pod api-6f8d4c7b59-t7xvw State: Waiting
Reason: ImagePullBackOff
Events:
Type Reason Age From Message
Normal Scheduled 3m12s default-scheduler Successfully assigned default/api-6f8d4c7b59-t7xvw to node-a
Normal Pulling 98s (x4 over 3m11s) kubelet Pulling image "registry.example.com/api:1.4.3"
Warning Failed 97s (x4 over 3m10s) kubelet Failed to pull image "registry.example.com/api:1.4.3": rpc error: code = NotFound desc = failed to pull and unpack image "registry.example.com/api:1.4.3": failed to resolve reference "registry.example.com/api:1.4.3": registry.example.com/api:1.4.3: not found
Warning Failed 97s (x4 over 3m10s) kubelet Error: ErrImagePull
Normal BackOff 20s (x6 over 3m9s) kubelet Back-off pulling image "registry.example.com/api:1.4.3"
Warning Failed 20s (x6 over 3m9s) kubelet Error: ImagePullBackOffWatch it happen
How ImagePullBackOff unfolds
Step 3 · The failure
Same request, same refusal, longer waits
Every retry asks for the same missing tag and gets the same answer. The kubelet's wait grows after each failure, up to a cap of 300 seconds. The container never starts, so RESTARTS stays at 0 and there are no logs.
- retry
- refused
- back-off
Step 3 · The failure
Same request, same refusal, longer waits
Every retry asks for the same missing tag and gets the same answer. The kubelet's wait grows after each failure, up to a cap of 300 seconds. The container never starts, so RESTARTS stays at 0 and there are no logs.
- retry
- refused
- back-off
Where it breaks: The registry is where the request is refused, usually because of what the pod asked for: a name or tag that does not exist, or no credentials. The kubelet reports ErrImagePull and then ImagePullBackOff while it waits to try again.
In short
What ImagePullBackOff means
ImagePullBackOff means the container has never run, because its image could not be downloaded. The kubelet asks the container runtime for the image, the runtime asks the registry, and the registry says no. The first failure is reported as ErrImagePull. After that the kubelet waits before each new attempt, with a delay that grows up to five minutes, and the pod shows ImagePullBackOff during the wait. The pod has been scheduled and the node is fine. Nothing inside your application is involved, so there are no container logs to read. The registry's own answer is in the pod's events, and it nearly always names the cause: the image or tag does not exist, the registry wants credentials the pod does not have, the node has hit a pull rate limit, or the image was not built for the node's CPU architecture.
In order
Diagnose in 60 seconds
- 1
kubectl get pod <pod>- Look for
- STATUS ImagePullBackOff or ErrImagePull, with RESTARTS at 0.
- What it tells you
- Rules in: the image never arrived. Rules out: scheduling, which already worked, and your application, which has not run yet.
- 2
kubectl describe pod <pod>- Look for
- The “Failed to pull image” event. Read the end of its message: it is the registry's own answer.
- What it tells you
- “not found” or “manifest unknown”: wrong name or tag. “unauthorized” or “pull access denied”: credentials. “toomanyrequests”: rate limit. “no match for platform”: wrong CPU architecture. A timeout or “no such host”: the node cannot reach the registry.
- 3
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].image}{"\n"}'- Look for
- The exact image reference the pod asks for: registry host, repository, tag or digest.
- What it tells you
- Rules in a typo, a tag that was never pushed, or a missing registry host that makes the node look on Docker Hub.
- 4
kubectl get pod <pod> -o jsonpath='{.spec.imagePullSecrets}{"\n"}'- Look for
- The names of the pull secrets attached to the pod. An empty result means it pulls anonymously.
- What it tells you
- Rules in missing credentials for a private registry.
- 5
kubectl get secret <pull-secret> -n <namespace>- Look for
- That the secret exists in the pod's own namespace and has type kubernetes.io/dockerconfigjson.
- What it tells you
- Rules in a secret that is missing, in another namespace, or of the wrong type. If it exists, its stored password or token may have expired.
Most common first
Root causes
1The image name or tag is wrong, or the tag was never pushed.
- Confirm
- The event ends in “not found” or “manifest unknown”.
- Fix now
- Correct the image reference in the Deployment, or push the missing tag.
- Prevent
- Have the pipeline deploy the exact tag or digest it just pushed, instead of typing it by hand.
2The registry is private and the pod has no valid credentials.
- Confirm
- The event says “unauthorized”, “authentication required” or “pull access denied”.
- Fix now
- Create a docker-registry Secret in the pod's namespace and name it under imagePullSecrets, or attach it to the pod's ServiceAccount.
- Prevent
- Attach the pull secret to the namespace's ServiceAccount so new workloads get it by default, and rotate tokens before they expire.
3The registry is rate-limiting the node's pulls.
- Confirm
- The event contains “toomanyrequests” or status 429.
- Fix now
- Wait for the limit to reset, or pull with an authenticated account that has a higher allowance.
- Prevent
- Mirror public images into your own registry, and avoid imagePullPolicy: Always where a fixed tag would do.
4The image has no build for the node's CPU architecture.
- Confirm
- The event says “no match for platform in manifest”. The node is arm64 and the image is amd64 only, or the reverse.
- Fix now
- Build and push a multi-architecture image, or schedule the pod onto nodes of the matching architecture.
- Prevent
- Build for every architecture your node pools use.
5The node cannot reach the registry: DNS, a firewall, a proxy or an untrusted certificate.
- Confirm
- The event shows a timeout, “no such host” or an x509 certificate error instead of an answer from the registry.
- Fix now
- Restore the network path from the node to the registry, or install the registry's CA certificate on the nodes.
- Prevent
- Test a pull from a node whenever network rules or registry certificates change.
The tempting wrong fix
Delete the pod and let it try again
Before / after
A common fix: give the pod credentials for a private registry
spec:
containers:
- name: api
image: registry.example.com/api:1.4.2
# no imagePullSecrets: the node pulls anonymously
# and the private registry answers “unauthorized”spec:
imagePullSecrets:
- name: registry-credentials # same namespace as the pod
containers:
- name: api
image: registry.example.com/api:1.4.2Create the Secret first with kubectl create secret docker-registry. If the event says “not found” instead of “unauthorized”, credentials are not the problem: fix the image name or tag.
In the real world
Depends on version and provider
Your turn
What would you do?
A pod is in ImagePullBackOff. Its events say: Failed to pull image “registry.example.com/api:1.4.2”: … pull access denied, repository does not exist or may require authorization. What should you check first?
Related errors
ImagePullBackOff questions
- What does ImagePullBackOff mean in Kubernetes?
- The kubelet could not pull the container image, so the container cannot start. It keeps trying, waiting longer between attempts, and shows ImagePullBackOff during each wait.
- What is the difference between ErrImagePull and ImagePullBackOff?
- They are two moments of the same problem. ErrImagePull is reported when a pull attempt has just failed. ImagePullBackOff is reported while the kubelet waits before the next attempt.
- How do I fix ImagePullBackOff?
- Run kubectl describe pod and read the “Failed to pull image” event. Then fix what it names: the image name or tag, the pull credentials, a registry rate limit, the image's architecture, or the node's network path to the registry.
Sources: Kubernetes docs: Images, ImagePullBackOff ↗ · Kubernetes docs: Images, image pull policy ↗ · Kubernetes docs: Pull an image from a private registry ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.