Skip to content
Breaks at Your container · Controller managerDegrades the serviceCommon

Deployment rollout failure

A rolling update has stopped partway: the new pods never become Ready, so the old ones are never removed.

What you'll seeIllustrative output
$ kubectl rollout status deployment/web
Waiting for deployment "web" rollout to finish: 2 out of 4 new replicas have been updated...
error: deployment "web" exceeded its progress deadline
$ kubectl get deploy,rs -l app=web
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/web   3/4     2            3           41d

NAME                             DESIRED   CURRENT   READY   AGE
replicaset.apps/web-6b7f9c5d8    3         3         3       9d
replicaset.apps/web-7c9d8f6b5    2         2         0       11m
$ kubectl describe deployment web
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    False   ProgressDeadlineExceeded
OldReplicaSets:  web-6b7f9c5d8 (3/3 replicas created)
NewReplicaSet:   web-7c9d8f6b5 (2/2 replicas created)

Watch it happen

How Deployment rollout failure unfolds

Step 3 · The failure

The new pods never become Ready

The v2 pods start, but their readiness checks keep failing. The controller is waiting for a Ready that never comes, so it removes nothing more: three v1 pods keep serving. After 600 seconds without progress the Deployment reports ProgressDeadlineExceeded. It does not roll back by itself.

  • not Ready
  • 0/1 report
  • old pods Ready

Where it breaks: The new version's pods are what fail: they never become Ready. The Deployment controller in the controller manager is working as designed when it stops: it will not take more old pods away until new ones can replace them.

In short

What Deployment rollout failure means

A rolling update replaces pods a few at a time. Two settings bound it: maxSurge, how many extra pods may exist, and maxUnavailable, how many may be missing; both default to 25%. The Deployment controller creates new pods up to the surge, and removes old ones only as new ones become Ready. That rule is the safety net. If the new version cannot become Ready, because the image is wrong, the container crashes, a probe fails or there is no room to schedule it, the rollout simply stops with most old pods still serving. Nothing is rolled back for you. After progressDeadlineSeconds without progress, 600 seconds by default, the Deployment's Progressing condition turns False with reason ProgressDeadlineExceeded, and kubectl rollout status exits with an error. The repair is to find why the new pods are not Ready, or to undo.

In order

Diagnose in 60 seconds

  1. 1
    kubectl rollout status deployment/<name>
    Look for
    Whether it finishes, keeps waiting, or ends with “exceeded its progress deadline”.
    What it tells you
    Rules in a stalled rollout, and tells you how many new replicas exist and how many are available.
  2. 2
    kubectl get rs -l app=<name>
    Look for
    Two ReplicaSets with pods. For the newest one, compare DESIRED, CURRENT and READY.
    What it tells you
    CURRENT below DESIRED: pods cannot be created (quota, admission). READY below CURRENT: pods exist but are not Ready. That is the usual case.
  3. 3
    kubectl get pods -l app=<name> --sort-by=.metadata.creationTimestamp
    Look for
    The STATUS of the newest pods: Pending, ImagePullBackOff, CrashLoopBackOff, or Running with 0/1.
    What it tells you
    The status names the real problem. From here the matching error page takes over.
  4. 4
    kubectl describe pod <new-pod>
    Look for
    The Events at the bottom: FailedScheduling, Failed to pull image, Back-off restarting, Readiness probe failed.
    What it tells you
    Gives the reason the new version is not Ready, which is the root cause of the stall.
  5. 5
    kubectl rollout undo deployment/<name>
    Look for
    The old ReplicaSet scaling back to the full replica count and the new one to zero.
    What it tells you
    Restores the last working version while you fix the new one. Add --to-revision=<n> to pick a specific revision from kubectl rollout history.

Most common first

Root causes

  1. 1The new pods run but fail their readiness probe: a bug, a missing dependency or setting, or a probe that no longer matches the app.

    Confirm
    New pods show Running with 0/1, and their events say “Readiness probe failed”.
    Fix now
    Undo the rollout, then correct the app or the probe.
    Prevent
    Run the same probes in a staging namespace before production, and keep them in step with the code.
  2. 2The new container crashes on start: CrashLoopBackOff.

    Confirm
    New pods show CrashLoopBackOff, and kubectl logs --previous shows why the process exits.
    Fix now
    Undo, then fix the configuration or code that makes it exit.
    Prevent
    Run an automated smoke test against the new image with production-like configuration in the pipeline, and release critical services as a canary to a small share of traffic first.
  3. 3The image cannot be pulled: a wrong tag, or missing registry credentials.

    Confirm
    New pods show ImagePullBackOff or ErrImagePull.
    Fix now
    Correct the image reference or the imagePullSecrets, or undo.
    Prevent
    Deploy by digest or by a tag the pipeline has just pushed and verified.
  4. 4There is no room for the surge pods: they stay Pending because the cluster or the namespace quota is full.

    Confirm
    New pods are Pending with FailedScheduling events, or the ReplicaSet reports FailedCreate because of a quota.
    Fix now
    Free capacity or add nodes, or allow the rollout to remove an old pod first by setting maxSurge 0 and maxUnavailable 1.
    Prevent
    Keep headroom for at least the surge, and remember that quotas count surge pods too.
  5. 5The app is just slow to become Ready and the deadline is shorter than a normal rollout step.

    Confirm
    The new pods do become Ready eventually, and the rollout completes after the error was reported.
    Fix now
    Nothing is broken. Raise progressDeadlineSeconds to fit the real start-up time.
    Prevent
    Measure how long one pod takes to become Ready and set the deadline with margin.

The tempting wrong fix

Delete the old pods to push the rollout along

The old pods are the only ones serving. The controller is holding them on purpose, because the new ones are not Ready. If you delete them, the old ReplicaSet recreates them, or worse, you are left with fewer working pods and a new version that still cannot take traffic. The stall is protecting you: fix or undo the new version instead.

Before / after

The most common fix: let the rollout tell you quickly, and never dip below capacity

Before
spec:
  replicas: 4
  # defaults: maxSurge 25%, maxUnavailable 25%
  # progressDeadlineSeconds: 600
  template:
    spec:
      containers:
        - name: web
          image: registry.example.com/web:2.0.0
          # no readinessProbe: a broken pod
          # counts as Ready the moment it starts
After
spec:
  replicas: 4
  progressDeadlineSeconds: 180
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0   # keep all 4 serving
  template:
    spec:
      containers:
        - name: web
          image: registry.example.com/web:2.0.0
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }

The readiness probe is what makes a rollout safe. Without one, a pod counts as Ready as soon as its container starts, and the controller will happily replace every working pod with a broken one.

In the real world

Depends on version and provider

The 25% defaults and the 600 second deadline are upstream and apply on every managed provider. The percentages are turned into pod counts by rounding maxSurge up and maxUnavailable down, so small Deployments behave differently: with one replica you get a surge of 1 and no unavailability. A paused Deployment does not count towards the deadline. Kubernetes only reports the stall; automatic rollback is something deployment tools add on top, such as Helm with --atomic, Argo Rollouts or Flagger, and each has its own timeouts.

Seen in a real outage

A rollout that stops partway leaves two versions running side by side. At Knight Capital in 2012, new code reached seven of eight servers, and the eighth kept running old code that reacted to the same switch in a very different way.

Break it: Seven of eight

Your turn

What would you do?

A rollout of a 4-replica Deployment has stalled: 3 old pods Ready, 2 new pods at 0/1, and the status shows ProgressDeadlineExceeded. What will Kubernetes do next if you leave it alone?

Deployment rollout failure questions

What does ProgressDeadlineExceeded mean?
The Deployment made no rollout progress for progressDeadlineSeconds, 600 seconds by default. Its Progressing condition is set to False with that reason. It is a report only: Kubernetes does not roll back or stop retrying.
Why is my Kubernetes rollout stuck?
Almost always because the new pods are not becoming Ready. The Deployment controller will not remove more old pods until they do. Check the newest pods' status and events: a failing readiness probe, a crash loop, an image that cannot be pulled, or no room to schedule.
How do I roll back a failed Deployment?
Run kubectl rollout undo deployment/<name>. It switches back to the previous revision. Use kubectl rollout history to list revisions and --to-revision to choose one.

Sources: Kubernetes docs: Deployments, failed Deployment ↗ · Kubernetes docs: Deployments, rolling update strategy ↗ · Kubernetes docs: Deployments, rolling back ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play