CPU throttling
The container used up its CPU quota for the current 100ms period, so the kernel paused it until the next one.
$ kubectl top pod api-6c7d8f9b5-kq2xw --containersPOD NAME CPU(cores) MEMORY(bytes) api-6c7d8f9b5-kq2xw api 287m 210Mi
$ kubectl exec api-6c7d8f9b5-kq2xw -- cat /sys/fs/cgroup/cpu.statusage_usec 81422310 user_usec 70113904 system_usec 11308406 nr_periods 48211 nr_throttled 19604 throttled_usec 912004211
$ kubectl get pod api-6c7d8f9b5-kq2xw -o jsonpath='{.spec.containers[0].resources}{"\n"}'{"limits":{"cpu":"500m"},"requests":{"cpu":"250m"}}Watch it happen
How CPU throttling unfolds
Step 3 · The failure
A burst spends it in a quarter of the period
Several threads work at once and burn the whole 50ms in the first quarter of the period. The container is then paused for the other three quarters. The next period is quiet, so average CPU looks low, yet every burst adds a long stall.
- CPU used / waiting
- answered late
- answered
Step 3 · The failure
A burst spends it in a quarter of the period
Several threads work at once and burn the whole 50ms in the first quarter of the period. The container is then paused for the other three quarters. The next period is quiet, so average CPU looks low, yet every burst adds a long stall.
- CPU used / waiting
- answered late
- answered
Where it breaks: The Linux kernel does the pausing, enforcing the CPU limit you set. Nothing is broken and nothing restarts, which is why it is easy to miss.
In short
What CPU throttling means
A CPU limit is not a speed limit. The kernel turns it into a budget of CPU time per period, 100 milliseconds by default. A limit of 500m means 50ms of CPU time in every 100ms. If the container spends that budget in the first part of a period, which a few busy threads can do in 15 or 20 milliseconds, the kernel stops running it until the next period begins. Requests that arrive in the meantime just wait. Averaged over a minute, CPU use can look modest, because quiet periods balance the busy ones, yet every busy period adds tens of milliseconds of delay. Nothing crashes and no event is recorded; the only direct evidence is the throttling counters in the container's cgroup. The request, by contrast, never causes throttling: it is what the scheduler reserves for the pod.
In order
Diagnose in 60 seconds
- 1
kubectl top pod <pod> --containers- Look for
- CPU well below the limit while users report slow responses. Needs metrics-server.
- What it tells you
- Rules out plain CPU exhaustion. A low average does not rule out throttling: it is averaged over many periods.
- 2
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources}{"\n"}'- Look for
- Whether a CPU limit is set, and how it compares with the request.
- What it tells you
- No CPU limit means no quota and no throttling of this kind. Look elsewhere, starting with a crowded node.
- 3
kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat- Look for
- nr_throttled against nr_periods, and throttled_usec. Run it twice a minute apart and compare. On cgroup v1 nodes the file is /sys/fs/cgroup/cpu/cpu.stat.
- What it tells you
- Rules in throttling when nr_throttled keeps growing. The share of throttled periods tells you how often requests hit a stall.
- 4
kubectl describe node <node>- Look for
- Allocated resources: the CPU requests and limits committed on the node, and its capacity.
- What it tells you
- Rules in a crowded node. Without limits, containers are still squeezed down towards their requests when the node is busy.
- 5
kubectl get pod <pod> -o jsonpath='{.status.qosClass}{"\n"}'- Look for
- Guaranteed, Burstable or BestEffort.
- What it tells you
- Tells you what you are changing: raising a limit above the request moves a Guaranteed pod to Burstable.
Most common first
Root causes
1The limit is too low for bursts: several threads spend the quota in a fraction of the period.
- Confirm
- nr_throttled rises steadily although average CPU is far below the limit, and latency has a long tail.
- Fix now
- Raise the CPU limit, or remove it and keep a realistic request.
- Prevent
- Size limits for peak concurrency inside one period, not for the one-minute average.
2The runtime starts as many worker threads as the node has cores, not as many as the limit allows.
- Confirm
- Thread or worker count matches the node's core count, and throttling starts as soon as load arrives.
- Fix now
- Set the worker or thread pool size from the container's CPU limit.
- Prevent
- Configure runtime parallelism explicitly in the image instead of letting it read the host.
3Start-up or periodic work (compilation, cache warm-up, garbage collection) needs far more CPU than steady serving.
- Confirm
- Throttling clusters at start-up or at regular intervals, and probes time out in those moments.
- Fix now
- Allow more CPU, or give start-up more time with a startup probe.
- Prevent
- Measure CPU during start and during background work, not only under steady load.
4The limit is simply lower than the sustained load.
- Confirm
- kubectl top shows usage sitting at the limit and nearly every period is throttled.
- Fix now
- Raise the limit and request, or add replicas.
- Prevent
- Scale on CPU utilisation with a HorizontalPodAutoscaler before the limit is reached.
The tempting wrong fix
Add more replicas
Before / after
The most common fix: a limit with room for bursts
resources:
requests:
cpu: 250m
limits:
cpu: 500m # 50ms of CPU time per 100ms periodresources:
requests:
cpu: 250m # what the scheduler reserves; unchanged
limits:
cpu: "1" # 100ms per period: bursts fit
# or leave the cpu limit out and rely on the requestThe request decides where the pod is scheduled and its share when the node is busy. Only the limit creates a quota. Keep the memory limit either way: memory cannot be throttled, only killed.
In the real world
Depends on version and provider
Your turn
What would you do?
An API has a CPU limit of 500m. kubectl top shows it at about 200m, yet one request in ten takes 80ms longer than the rest. What is the most likely explanation?
Related errors
CPU throttling questions
- What is CPU throttling in Kubernetes?
- It is the kernel pausing a container that has used its CPU quota for the current period. A CPU limit is enforced as CPU time per 100ms period by default, and once that time is spent the container does not run again until the next period.
- Why is my pod throttled when CPU usage is low?
- Because usage is averaged over many periods. A short burst on several threads can spend the whole quota within one period and be paused, while the quiet periods around it keep the average low.
- Should I remove CPU limits?
- It is a common and reasonable choice for latency-sensitive services, as long as every container has a realistic CPU request. Requests still protect each pod's share when the node is busy. Keep memory limits, which work differently.
Sources: Kubernetes docs: Resource management for pods and containers ↗ · Kubernetes docs: Assign CPU resources to containers and pods ↗ · Kubernetes docs: Control CPU management policies on the node ↗ · Report a problem
Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.