Skip to content
Breaks at Linux kernel · Your containerDegrades the serviceCommon

CPU throttling

The container used up its CPU quota for the current 100ms period, so the kernel paused it until the next one.

What you'll seeIllustrative output
$ kubectl top pod api-6c7d8f9b5-kq2xw --containers
POD                   NAME   CPU(cores)   MEMORY(bytes)
api-6c7d8f9b5-kq2xw   api    287m         210Mi
$ kubectl exec api-6c7d8f9b5-kq2xw -- cat /sys/fs/cgroup/cpu.stat
usage_usec 81422310
user_usec 70113904
system_usec 11308406
nr_periods 48211
nr_throttled 19604
throttled_usec 912004211
$ kubectl get pod api-6c7d8f9b5-kq2xw -o jsonpath='{.spec.containers[0].resources}{"\n"}'
{"limits":{"cpu":"500m"},"requests":{"cpu":"250m"}}

Watch it happen

How CPU throttling unfolds

Step 3 · The failure

A burst spends it in a quarter of the period

Several threads work at once and burn the whole 50ms in the first quarter of the period. The container is then paused for the other three quarters. The next period is quiet, so average CPU looks low, yet every burst adds a long stall.

  • CPU used / waiting
  • answered late
  • answered

Where it breaks: The Linux kernel does the pausing, enforcing the CPU limit you set. Nothing is broken and nothing restarts, which is why it is easy to miss.

In short

What CPU throttling means

A CPU limit is not a speed limit. The kernel turns it into a budget of CPU time per period, 100 milliseconds by default. A limit of 500m means 50ms of CPU time in every 100ms. If the container spends that budget in the first part of a period, which a few busy threads can do in 15 or 20 milliseconds, the kernel stops running it until the next period begins. Requests that arrive in the meantime just wait. Averaged over a minute, CPU use can look modest, because quiet periods balance the busy ones, yet every busy period adds tens of milliseconds of delay. Nothing crashes and no event is recorded; the only direct evidence is the throttling counters in the container's cgroup. The request, by contrast, never causes throttling: it is what the scheduler reserves for the pod.

In order

Diagnose in 60 seconds

  1. 1
    kubectl top pod <pod> --containers
    Look for
    CPU well below the limit while users report slow responses. Needs metrics-server.
    What it tells you
    Rules out plain CPU exhaustion. A low average does not rule out throttling: it is averaged over many periods.
  2. 2
    kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources}{"\n"}'
    Look for
    Whether a CPU limit is set, and how it compares with the request.
    What it tells you
    No CPU limit means no quota and no throttling of this kind. Look elsewhere, starting with a crowded node.
  3. 3
    kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat
    Look for
    nr_throttled against nr_periods, and throttled_usec. Run it twice a minute apart and compare. On cgroup v1 nodes the file is /sys/fs/cgroup/cpu/cpu.stat.
    What it tells you
    Rules in throttling when nr_throttled keeps growing. The share of throttled periods tells you how often requests hit a stall.
  4. 4
    kubectl describe node <node>
    Look for
    Allocated resources: the CPU requests and limits committed on the node, and its capacity.
    What it tells you
    Rules in a crowded node. Without limits, containers are still squeezed down towards their requests when the node is busy.
  5. 5
    kubectl get pod <pod> -o jsonpath='{.status.qosClass}{"\n"}'
    Look for
    Guaranteed, Burstable or BestEffort.
    What it tells you
    Tells you what you are changing: raising a limit above the request moves a Guaranteed pod to Burstable.

Most common first

Root causes

  1. 1The limit is too low for bursts: several threads spend the quota in a fraction of the period.

    Confirm
    nr_throttled rises steadily although average CPU is far below the limit, and latency has a long tail.
    Fix now
    Raise the CPU limit, or remove it and keep a realistic request.
    Prevent
    Size limits for peak concurrency inside one period, not for the one-minute average.
  2. 2The runtime starts as many worker threads as the node has cores, not as many as the limit allows.

    Confirm
    Thread or worker count matches the node's core count, and throttling starts as soon as load arrives.
    Fix now
    Set the worker or thread pool size from the container's CPU limit.
    Prevent
    Configure runtime parallelism explicitly in the image instead of letting it read the host.
  3. 3Start-up or periodic work (compilation, cache warm-up, garbage collection) needs far more CPU than steady serving.

    Confirm
    Throttling clusters at start-up or at regular intervals, and probes time out in those moments.
    Fix now
    Allow more CPU, or give start-up more time with a startup probe.
    Prevent
    Measure CPU during start and during background work, not only under steady load.
  4. 4The limit is simply lower than the sustained load.

    Confirm
    kubectl top shows usage sitting at the limit and nearly every period is throttled.
    Fix now
    Raise the limit and request, or add replicas.
    Prevent
    Scale on CPU utilisation with a HorizontalPodAutoscaler before the limit is reached.

The tempting wrong fix

Add more replicas

Each replica has the same quota and the same bursts, so each one still stalls inside its own periods. Extra pods lower the load per pod a little and cost a lot. The stall is caused by the limit on one container, so that is the number to change.

Before / after

The most common fix: a limit with room for bursts

Before
resources:
  requests:
    cpu: 250m
  limits:
    cpu: 500m    # 50ms of CPU time per 100ms period
After
resources:
  requests:
    cpu: 250m    # what the scheduler reserves; unchanged
  limits:
    cpu: "1"     # 100ms per period: bursts fit
    # or leave the cpu limit out and rely on the request

The request decides where the pod is scheduled and its share when the node is busy. Only the limit creates a quota. Keep the memory limit either way: memory cannot be throttled, only killed.

In the real world

Depends on version and provider

The 100ms period is the kubelet's default (cpuCFSQuotaPeriod) and quota enforcement can be switched off for a whole node (cpuCFSQuota), so check with whoever runs the nodes. Managed providers generally keep the defaults. With the static CPU manager policy, Guaranteed pods that request whole CPUs get exclusive cores, and newer kubelets can exempt them from quota. The counter file differs by cgroup version: cpu.stat under /sys/fs/cgroup on v2, under /sys/fs/cgroup/cpu on v1.

Your turn

What would you do?

An API has a CPU limit of 500m. kubectl top shows it at about 200m, yet one request in ten takes 80ms longer than the rest. What is the most likely explanation?

CPU throttling questions

What is CPU throttling in Kubernetes?
It is the kernel pausing a container that has used its CPU quota for the current period. A CPU limit is enforced as CPU time per 100ms period by default, and once that time is spent the container does not run again until the next period.
Why is my pod throttled when CPU usage is low?
Because usage is averaged over many periods. A short burst on several threads can spend the whole quota within one period and be paused, while the quiet periods around it keep the average low.
Should I remove CPU limits?
It is a common and reasonable choice for latency-sensitive services, as long as every container has a realistic CPU request. Requests still protect each pod's share when the node is busy. Keep memory limits, which work differently.

Sources: Kubernetes docs: Resource management for pods and containers ↗ · Kubernetes docs: Assign CPU resources to containers and pods ↗ · Kubernetes docs: Control CPU management policies on the node ↗ · Report a problem

Diagrams are our own simulations. Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play