Skip to content

Walkthrough

Production incident: a Spring Boot API becomes slow

A Java microservice on Kubernetes. Response time has gone from 200ms to 5 seconds. CPU is at 95%, the pods are healthy, there are no exceptions in the logs, and traffic is four times normal. Work through it in order: each step rules something out.

The investigation

  1. 1 · Check pod health

    Rule out the loud failures first: crashes, restarts and pods that are not Ready.

    RunIllustrative output
    $ kubectl get pods -l app=payment-service
    NAME                               READY   STATUS    RESTARTS   AGE
    payment-service-6c7d8f9b5-4kq2x   1/1     Running   0          3d
    payment-service-6c7d8f9b5-9wd7m   1/1     Running   0          3d
    payment-service-6c7d8f9b5-tz8pl   1/1     Running   0          3d

    What it tells you: All Running, all Ready, zero restarts. Nothing is crashing or being killed, so this is not a crash loop, an OOM kill or a probe problem. The pods are working, only slowly.

    CrashLoopBackOff
  2. 2 · Check CPU and memory

    See which resource the pods are pressed against.

    RunIllustrative output
    $ kubectl top pods -l app=payment-service
    NAME                               CPU(cores)   MEMORY(bytes)
    payment-service-6c7d8f9b5-4kq2x   498m         612Mi
    payment-service-6c7d8f9b5-9wd7m   500m         598Mi
    payment-service-6c7d8f9b5-tz8pl   497m         620Mi

    What it tells you: Memory is steady and well under the limit. CPU sits at almost exactly 500m in every pod. A flat line at a round number is a ceiling, not a coincidence: the CPU limit is 500m.

    OOMKilled
  3. 3 · Check throttling

    Confirm that the kernel is pausing the containers at their CPU limit.

    RunIllustrative output
    $ kubectl exec payment-service-6c7d8f9b5-4kq2x -- cat /sys/fs/cgroup/cpu.stat
    usage_usec 81422910334
    nr_periods 48211
    nr_throttled 39604
    throttled_usec 2912004211

    What it tells you: The container was throttled in most periods. Requests are not failing; they are waiting for the next slice of CPU time, which is where the seconds go. The same figure is available as a throttling metric in Prometheus and Grafana.

    CPU throttling
  4. 4 · Check the autoscaler

    With four times the traffic, more replicas should have appeared. Find out why they did not.

    RunIllustrative output
    $ kubectl describe hpa payment-service
    Metrics:                       ( current / target )
      resource cpu on pods (as a percentage of request):  99% (497m) / 70%
    Min replicas:                  3
    Max replicas:                  3
    Conditions:
      Type            Status  Reason
      ScalingLimited  True    TooManyReplicas

    What it tells you: The autoscaler sees 99% against a 70% target and wants more pods, but maxReplicas is 3 and it is already there. Scaling is working as configured; the configuration is the limit.

    HPA not scaling
  5. 5 · Check the application itself

    Make sure the CPU is going to real work, and that nothing downstream is the true bottleneck.

    What it tells you: Look at JVM garbage collection time, a thread dump, connection pool usage and database latency. Here GC is normal, threads are busy handling requests and the database answers in a few milliseconds: the service is simply short of CPU for this much traffic. If instead the threads were all waiting on a connection pool or a slow query, adding CPU or replicas would not help.

    HTTP 504: when a slow backend becomes a timeout
  6. 6 · Mitigate

    Restore latency first, then fix the cause properly.

    RunIllustrative output
    $ kubectl patch hpa payment-service --patch '{"spec":{"maxReplicas":12}}'
    horizontalpodautoscaler.autoscaling/payment-service patched

    What it tells you: Raising maxReplicas lets the autoscaler add pods, provided the cluster has room for them. Latency returns to normal as load spreads out. Afterwards, review the CPU limit: a limit equal to the request leaves no room for bursts.

    Pending pod: when the new replicas have nowhere to go

Expected outcome

API latency is back to normal, and the underlying bottleneck has a name: the pods were throttled at a 500m CPU limit while the autoscaler was capped at three replicas. The lasting fixes are a higher maxReplicas, CPU requests and limits sized from measurements, and an alert on throttling so the next traffic spike is visible before users notice.

The configuration that caused it

Example
resources:
  requests:
    cpu: "500m"
  limits:
    cpu: "500m"    # half a core, no room to burst
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 3
  maxReplicas: 3    # nowhere to scale to

CPU limits throttle; memory limits kill. A pod that is too slow and never restarts points at CPU, while a pod that restarts with exit code 137 points at memory.

Sources: Kubernetes docs: Resource management for pods and containers ↗ · Kubernetes docs: Horizontal pod autoscaling ↗ · Report a problem

Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play