HPA not scaling
Load rises but the HorizontalPodAutoscaler leaves the replica count where it is.
$ kubectl get hpaNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE api Deployment/api cpu: <unknown>/70% 1 10 1 12m
$ kubectl describe hpa apiConditions: Type Status Reason Message AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu in container api of Pod api-6c7d8f9b5-kq2xw Events: Type Reason Age From Message Warning FailedGetResourceMetric 8s (x47 over 12m) horizontal-pod-autoscaler failed to get cpu utilization: missing request for cpu in container api of Pod api-6c7d8f9b5-kq2xw Warning FailedComputeMetricsReplicas 8s (x47 over 12m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value
$ kubectl get deployment api -o jsonpath='{.spec.template.spec.containers[*].resources}{"\n"}'{}Watch it happen
How HPA not scaling unfolds
Step 3 · The failure
No request, no percentage, no scaling
Same load, but the pod spec has no CPU request. Usage still arrives, and the HPA has nothing to divide it by. Utilisation is <unknown>, so every sync ends with no decision. Replicas stay at 1 while requests fail.
- usage
- unknown / failed request
- answered
Step 3 · The failure
No request, no percentage, no scaling
Same load, but the pod spec has no CPU request. Usage still arrives, and the HPA has nothing to divide it by. Utilisation is <unknown>, so every sync ends with no decision. Replicas stay at 1 while requests fail.
- usage
- unknown / failed request
- answered
Where it breaks: The HPA controller is where it shows: it cannot compute a utilisation, or has hit its ceiling. The cause is usually elsewhere: pods without CPU requests, a missing metrics-server, or a cluster with no room for more pods.
In short
What HPA not scaling means
A HorizontalPodAutoscaler runs a simple loop. Every 15 seconds it reads a metric for the pods of its target, compares it with the target value, and writes a new replica count if the difference is large enough. For CPU utilisation, the metric is usage divided by each container's CPU request. That is where it most often breaks. If any container has no CPU request, utilisation is undefined, kubectl shows the target as <unknown>, and the autoscaler takes no action at all. The same happens when nothing serves the metrics API, because metrics-server is not installed or not healthy. Other cases look like failures but are the autoscaler working as configured: the count has reached maxReplicas, new pods are created but stay Pending because the cluster is full, or it is waiting out the five-minute stabilisation window before scaling down. kubectl describe hpa says which.
In order
Diagnose in 60 seconds
- 1
kubectl get hpa <name>- Look for
- TARGETS: a percentage, or <unknown>. Then REPLICAS against MINPODS and MAXPODS.
- What it tells you
- <unknown> means the metric cannot be computed. A real number with REPLICAS equal to MAXPODS means it has scaled as far as it is allowed.
- 2
kubectl describe hpa <name>- Look for
- Conditions: ScalingActive False with a reason, or ScalingLimited True with TooManyReplicas. Then the Events.
- What it tells you
- The message names the cause in words: a missing request, no metrics returned, or the replica limit.
- 3
kubectl top pods- Look for
- Whether CPU figures come back, or an error that the Metrics API is not available.
- What it tells you
- Rules in a missing or broken metrics-server. If figures appear, metrics are fine and the problem is in how they are used.
- 4
kubectl get apiservice v1beta1.metrics.k8s.io- Look for
- AVAILABLE True, or False with a reason such as MissingEndpoints or FailedDiscoveryCheck.
- What it tells you
- Confirms whether anything is serving the metrics API, and whether the API server can reach it.
- 5
kubectl get deployment <name> -o jsonpath='{.spec.template.spec.containers[*].resources}{"\n"}'- Look for
- A requests.cpu entry for every container, sidecars included.
- What it tells you
- Rules in missing requests: one container without a request makes utilisation undefined for the whole pod.
- 6
kubectl get pods -l app=<name>- Look for
- New replicas stuck in Pending.
- What it tells you
- The HPA did scale; the cluster has no room. Continue on the Pending pod page.
Most common first
Root causes
1A container in the pod has no CPU request, so utilisation cannot be computed.
- Confirm
- TARGETS shows <unknown> and events say “missing request for cpu in container …”.
- Fix now
- Add resources.requests.cpu to every container in the pod template, injected sidecars included.
- Prevent
- Require requests with a LimitRange default or an admission policy.
2Nothing serves the metrics API: metrics-server is not installed, is crash-looping, or cannot reach the kubelets.
- Confirm
- kubectl top fails, and the v1beta1.metrics.k8s.io APIService is not Available.
- Fix now
- Install or repair metrics-server and check its own logs for TLS or network errors.
- Prevent
- Treat metrics-server as a critical add-on and alert when its APIService is unavailable.
3The replica count has reached maxReplicas.
- Confirm
- REPLICAS equals MAXPODS and the ScalingLimited condition is True with reason TooManyReplicas.
- Fix now
- Raise maxReplicas if the cluster and the systems behind the service can take it.
- Prevent
- Alert when an autoscaler sits at its maximum for more than a few minutes.
4The HPA scaled, but the new pods are Pending because no node has room.
- Confirm
- The Deployment's desired count went up and the extra pods show Pending with FailedScheduling events.
- Fix now
- Add nodes, or free capacity. Check that the cluster autoscaler is allowed to grow.
- Prevent
- Keep headroom, and size pod requests so that new replicas fit on existing nodes.
5It is waiting on purpose: the scale-down stabilisation window, or a change smaller than the tolerance.
- Confirm
- Load has dropped, the metric is below target, and replicas fall about five minutes later.
- Fix now
- Nothing is broken. Shorten behavior.scaleDown.stabilizationWindowSeconds only if the delay costs you.
- Prevent
- Know the defaults: 300 seconds before scaling down, none before scaling up, and a 10% tolerance.
6The HPA points at the wrong thing: a scaleTargetRef name or kind that does not match the workload.
- Confirm
- The AbleToScale condition is False with FailedGetScale.
- Fix now
- Correct scaleTargetRef to the Deployment or StatefulSet that actually runs the pods.
- Prevent
- Create the HPA in the same manifest or chart as its workload.
The tempting wrong fix
Lower the CPU target so it scales sooner
Before / after
The most common fix: a CPU request on every container
containers:
- name: api
image: registry.example.com/api:1.4.2
# no resources: utilisation is undefined,
# so the HPA target reads <unknown>containers:
- name: api
image: registry.example.com/api:1.4.2
resources:
requests:
cpu: 100m # utilisation = usage ÷ this
memory: 128MiUtilisation targets are a percentage of the request, not of the limit or of a core. A request set far too high makes the percentage look small and the HPA slow to react; set it close to what one pod normally uses.
In the real world
Depends on version and provider
Seen in a real outage
An autoscaler only knows the number it is given. At Slack in 2021 a network bottleneck left web servers waiting, so their CPU fell, and CPU-based autoscaling removed servers in the middle of an overload.
Break it: Overloaded, yet idleYour turn
What would you do?
kubectl get hpa shows TARGETS “cpu: <unknown>/70%”. kubectl top pods prints CPU figures normally. What should you check next?
Related errors
HPA not scaling questions
- Why does my HPA show <unknown> for its target?
- The autoscaler cannot compute the metric. For CPU utilisation the usual reasons are a container without a CPU request, or no working metrics-server. kubectl describe hpa gives the exact reason in its conditions and events.
- How often does the HPA check metrics?
- Every 15 seconds by default. It scales up as soon as the metric calls for it, and waits through a stabilisation window, 300 seconds by default, before scaling down.
- Why is my HPA not scaling down?
- Usually it is waiting. By default it keeps the highest replica count recommended during the last five minutes, to avoid flapping. It also ignores changes within 10% of the target and never goes below minReplicas.
Sources: Kubernetes docs: Horizontal pod autoscaling, how it works ↗ · Kubernetes docs: Horizontal pod autoscaling, algorithm details ↗ · Kubernetes docs: Resource metrics pipeline ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.