Skip to content
Breaks at HPA controller · metrics-serverDegrades the serviceCommon

HPA not scaling

Load rises but the HorizontalPodAutoscaler leaves the replica count where it is.

What you'll seeIllustrative output
$ kubectl get hpa
NAME   REFERENCE        TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
api    Deployment/api   cpu: <unknown>/70%   1         10        1          12m
$ kubectl describe hpa api
Conditions:
  Type           Status  Reason                   Message
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu in container api of Pod api-6c7d8f9b5-kq2xw
Events:
  Type     Reason                        Age                From                       Message
  Warning  FailedGetResourceMetric       8s (x47 over 12m)  horizontal-pod-autoscaler  failed to get cpu utilization: missing request for cpu in container api of Pod api-6c7d8f9b5-kq2xw
  Warning  FailedComputeMetricsReplicas  8s (x47 over 12m)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value
$ kubectl get deployment api -o jsonpath='{.spec.template.spec.containers[*].resources}{"\n"}'
{}

Watch it happen

How HPA not scaling unfolds

Step 3 · The failure

No request, no percentage, no scaling

Same load, but the pod spec has no CPU request. Usage still arrives, and the HPA has nothing to divide it by. Utilisation is <unknown>, so every sync ends with no decision. Replicas stay at 1 while requests fail.

  • usage
  • unknown / failed request
  • answered

Where it breaks: The HPA controller is where it shows: it cannot compute a utilisation, or has hit its ceiling. The cause is usually elsewhere: pods without CPU requests, a missing metrics-server, or a cluster with no room for more pods.

In short

What HPA not scaling means

A HorizontalPodAutoscaler runs a simple loop. Every 15 seconds it reads a metric for the pods of its target, compares it with the target value, and writes a new replica count if the difference is large enough. For CPU utilisation, the metric is usage divided by each container's CPU request. That is where it most often breaks. If any container has no CPU request, utilisation is undefined, kubectl shows the target as <unknown>, and the autoscaler takes no action at all. The same happens when nothing serves the metrics API, because metrics-server is not installed or not healthy. Other cases look like failures but are the autoscaler working as configured: the count has reached maxReplicas, new pods are created but stay Pending because the cluster is full, or it is waiting out the five-minute stabilisation window before scaling down. kubectl describe hpa says which.

In order

Diagnose in 60 seconds

  1. 1
    kubectl get hpa <name>
    Look for
    TARGETS: a percentage, or <unknown>. Then REPLICAS against MINPODS and MAXPODS.
    What it tells you
    <unknown> means the metric cannot be computed. A real number with REPLICAS equal to MAXPODS means it has scaled as far as it is allowed.
  2. 2
    kubectl describe hpa <name>
    Look for
    Conditions: ScalingActive False with a reason, or ScalingLimited True with TooManyReplicas. Then the Events.
    What it tells you
    The message names the cause in words: a missing request, no metrics returned, or the replica limit.
  3. 3
    kubectl top pods
    Look for
    Whether CPU figures come back, or an error that the Metrics API is not available.
    What it tells you
    Rules in a missing or broken metrics-server. If figures appear, metrics are fine and the problem is in how they are used.
  4. 4
    kubectl get apiservice v1beta1.metrics.k8s.io
    Look for
    AVAILABLE True, or False with a reason such as MissingEndpoints or FailedDiscoveryCheck.
    What it tells you
    Confirms whether anything is serving the metrics API, and whether the API server can reach it.
  5. 5
    kubectl get deployment <name> -o jsonpath='{.spec.template.spec.containers[*].resources}{"\n"}'
    Look for
    A requests.cpu entry for every container, sidecars included.
    What it tells you
    Rules in missing requests: one container without a request makes utilisation undefined for the whole pod.
  6. 6
    kubectl get pods -l app=<name>
    Look for
    New replicas stuck in Pending.
    What it tells you
    The HPA did scale; the cluster has no room. Continue on the Pending pod page.

Most common first

Root causes

  1. 1A container in the pod has no CPU request, so utilisation cannot be computed.

    Confirm
    TARGETS shows <unknown> and events say “missing request for cpu in container …”.
    Fix now
    Add resources.requests.cpu to every container in the pod template, injected sidecars included.
    Prevent
    Require requests with a LimitRange default or an admission policy.
  2. 2Nothing serves the metrics API: metrics-server is not installed, is crash-looping, or cannot reach the kubelets.

    Confirm
    kubectl top fails, and the v1beta1.metrics.k8s.io APIService is not Available.
    Fix now
    Install or repair metrics-server and check its own logs for TLS or network errors.
    Prevent
    Treat metrics-server as a critical add-on and alert when its APIService is unavailable.
  3. 3The replica count has reached maxReplicas.

    Confirm
    REPLICAS equals MAXPODS and the ScalingLimited condition is True with reason TooManyReplicas.
    Fix now
    Raise maxReplicas if the cluster and the systems behind the service can take it.
    Prevent
    Alert when an autoscaler sits at its maximum for more than a few minutes.
  4. 4The HPA scaled, but the new pods are Pending because no node has room.

    Confirm
    The Deployment's desired count went up and the extra pods show Pending with FailedScheduling events.
    Fix now
    Add nodes, or free capacity. Check that the cluster autoscaler is allowed to grow.
    Prevent
    Keep headroom, and size pod requests so that new replicas fit on existing nodes.
  5. 5It is waiting on purpose: the scale-down stabilisation window, or a change smaller than the tolerance.

    Confirm
    Load has dropped, the metric is below target, and replicas fall about five minutes later.
    Fix now
    Nothing is broken. Shorten behavior.scaleDown.stabilizationWindowSeconds only if the delay costs you.
    Prevent
    Know the defaults: 300 seconds before scaling down, none before scaling up, and a 10% tolerance.
  6. 6The HPA points at the wrong thing: a scaleTargetRef name or kind that does not match the workload.

    Confirm
    The AbleToScale condition is False with FailedGetScale.
    Fix now
    Correct scaleTargetRef to the Deployment or StatefulSet that actually runs the pods.
    Prevent
    Create the HPA in the same manifest or chart as its workload.

The tempting wrong fix

Lower the CPU target so it scales sooner

The target is not the problem when the metric is <unknown> or the count is at maxReplicas: 50% of an unknown number is still unknown. A lower target only makes a working autoscaler run more pods than it needs. Read the conditions in kubectl describe hpa first; they say what is blocking it.

Before / after

The most common fix: a CPU request on every container

Before
containers:
  - name: api
    image: registry.example.com/api:1.4.2
    # no resources: utilisation is undefined,
    # so the HPA target reads <unknown>
After
containers:
  - name: api
    image: registry.example.com/api:1.4.2
    resources:
      requests:
        cpu: 100m      # utilisation = usage ÷ this
        memory: 128Mi

Utilisation targets are a percentage of the request, not of the limit or of a core. A request set far too high makes the percentage look small and the HPA slow to react; set it close to what one pod normally uses.

In the real world

Depends on version and provider

metrics-server is not part of Kubernetes itself. GKE and AKS ship it; on EKS and on clusters built with kubeadm you usually install it yourself, and on kubeadm it often needs kubelet certificate settings sorted out before it works. The 15 second sync period, the 300 second scale-down window and the 10% tolerance are upstream defaults; the sync period is a control plane flag you cannot change on managed services, while the window and, on recent versions, the tolerance can be set per HPA under behavior. To scale on one container and ignore sidecars, use the ContainerResource metric type.

Seen in a real outage

An autoscaler only knows the number it is given. At Slack in 2021 a network bottleneck left web servers waiting, so their CPU fell, and CPU-based autoscaling removed servers in the middle of an overload.

Break it: Overloaded, yet idle

Your turn

What would you do?

kubectl get hpa shows TARGETS “cpu: <unknown>/70%”. kubectl top pods prints CPU figures normally. What should you check next?

HPA not scaling questions

Why does my HPA show <unknown> for its target?
The autoscaler cannot compute the metric. For CPU utilisation the usual reasons are a container without a CPU request, or no working metrics-server. kubectl describe hpa gives the exact reason in its conditions and events.
How often does the HPA check metrics?
Every 15 seconds by default. It scales up as soon as the metric calls for it, and waits through a stabilisation window, 300 seconds by default, before scaling down.
Why is my HPA not scaling down?
Usually it is waiting. By default it keeps the highest replica count recommended during the last five minutes, to avoid flapping. It also ignores changes within 10% of the target and never goes below minReplicas.

Sources: Kubernetes docs: Horizontal pod autoscaling, how it works ↗ · Kubernetes docs: Horizontal pod autoscaling, algorithm details ↗ · Kubernetes docs: Resource metrics pipeline ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play