Walkthrough
Production incident: a Spring Boot API becomes slow
A Java microservice on Kubernetes. Response time has gone from 200ms to 5 seconds. CPU is at 95%, the pods are healthy, there are no exceptions in the logs, and traffic is four times normal. Work through it in order: each step rules something out.
The investigation
1 · Check pod health
Rule out the loud failures first: crashes, restarts and pods that are not Ready.
RunIllustrative output$ kubectl get pods -l app=payment-serviceNAME READY STATUS RESTARTS AGE payment-service-6c7d8f9b5-4kq2x 1/1 Running 0 3d payment-service-6c7d8f9b5-9wd7m 1/1 Running 0 3d payment-service-6c7d8f9b5-tz8pl 1/1 Running 0 3d
What it tells you: All Running, all Ready, zero restarts. Nothing is crashing or being killed, so this is not a crash loop, an OOM kill or a probe problem. The pods are working, only slowly.
CrashLoopBackOff2 · Check CPU and memory
See which resource the pods are pressed against.
RunIllustrative output$ kubectl top pods -l app=payment-serviceNAME CPU(cores) MEMORY(bytes) payment-service-6c7d8f9b5-4kq2x 498m 612Mi payment-service-6c7d8f9b5-9wd7m 500m 598Mi payment-service-6c7d8f9b5-tz8pl 497m 620Mi
What it tells you: Memory is steady and well under the limit. CPU sits at almost exactly 500m in every pod. A flat line at a round number is a ceiling, not a coincidence: the CPU limit is 500m.
OOMKilled3 · Check throttling
Confirm that the kernel is pausing the containers at their CPU limit.
RunIllustrative output$ kubectl exec payment-service-6c7d8f9b5-4kq2x -- cat /sys/fs/cgroup/cpu.statusage_usec 81422910334 nr_periods 48211 nr_throttled 39604 throttled_usec 2912004211
What it tells you: The container was throttled in most periods. Requests are not failing; they are waiting for the next slice of CPU time, which is where the seconds go. The same figure is available as a throttling metric in Prometheus and Grafana.
CPU throttling4 · Check the autoscaler
With four times the traffic, more replicas should have appeared. Find out why they did not.
RunIllustrative output$ kubectl describe hpa payment-serviceMetrics: ( current / target ) resource cpu on pods (as a percentage of request): 99% (497m) / 70% Min replicas: 3 Max replicas: 3 Conditions: Type Status Reason ScalingLimited True TooManyReplicas
What it tells you: The autoscaler sees 99% against a 70% target and wants more pods, but maxReplicas is 3 and it is already there. Scaling is working as configured; the configuration is the limit.
HPA not scaling5 · Check the application itself
Make sure the CPU is going to real work, and that nothing downstream is the true bottleneck.
What it tells you: Look at JVM garbage collection time, a thread dump, connection pool usage and database latency. Here GC is normal, threads are busy handling requests and the database answers in a few milliseconds: the service is simply short of CPU for this much traffic. If instead the threads were all waiting on a connection pool or a slow query, adding CPU or replicas would not help.
HTTP 504: when a slow backend becomes a timeout6 · Mitigate
Restore latency first, then fix the cause properly.
RunIllustrative output$ kubectl patch hpa payment-service --patch '{"spec":{"maxReplicas":12}}'horizontalpodautoscaler.autoscaling/payment-service patched
What it tells you: Raising maxReplicas lets the autoscaler add pods, provided the cluster has room for them. Latency returns to normal as load spreads out. Afterwards, review the CPU limit: a limit equal to the request leaves no room for bursts.
Pending pod: when the new replicas have nowhere to go
Expected outcome
API latency is back to normal, and the underlying bottleneck has a name: the pods were throttled at a 500m CPU limit while the autoscaler was capped at three replicas. The lasting fixes are a higher maxReplicas, CPU requests and limits sized from measurements, and an alert on throttling so the next traffic spike is visible before users notice.
The configuration that caused it
resources:
requests:
cpu: "500m"
limits:
cpu: "500m" # half a core, no room to burst
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
minReplicas: 3
maxReplicas: 3 # nowhere to scale toCPU limits throttle; memory limits kill. A pod that is too slow and never restarts points at CPU, while a pod that restarts with exit code 137 points at memory.
Keep going
Sources: Kubernetes docs: Resource management for pods and containers ↗ · Kubernetes docs: Horizontal pod autoscaling ↗ · Report a problem
Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.