Cheat sheet
Essential kubectl troubleshooting commands
Fourteen commands cover most production investigations. Each row says what the command is for and links to the problem it most often uncovers. Add -n <namespace> to any of them.
Pods: what is running, and why not
kubectl get podsList pods with READY, STATUS and RESTARTS. The first look at any problem.
CrashLoopBackOffkubectl describe pod PODState, last state with exit code, probe settings and the events for one pod.
Pending podkubectl logs PODOutput of the current container.
Liveness probe failurekubectl logs POD --previousOutput of the container instance that ended before the last restart. This is where a crash explains itself.
CrashLoopBackOffkubectl get events --sort-by=.metadata.creationTimestampRecent events across the namespace in time order: scheduling, pulls, probes, kills.
ImagePullBackOff
Resources: memory and CPU
kubectl top podsCurrent CPU and memory per pod. Needs metrics-server. Add --containers for each container.
OOMKilledkubectl top nodesCurrent CPU and memory per node, to see whether the machines are full.
CPU throttlingkubectl get nodesNode health: Ready, NotReady or SchedulingDisabled.
Node NotReady
Traffic: Services and their endpoints
kubectl get svcList Services with their cluster IP and ports.
Service has no endpointskubectl get endpointslicesThe pod addresses a Service actually sends traffic to. Empty means no pod is selected or none is Ready.
Readiness probe failure
Releases and scaling
kubectl get deploymentsREADY, UP-TO-DATE and AVAILABLE counts for each Deployment.
Deployment rollout failurekubectl rollout status deploy/APPFollow a rollout until it finishes or reports that it has stalled.
Deployment rollout failurekubectl rollout undo deploy/APPReturn to the previous revision. The fastest safe move when a release is the cause.
Deployment rollout failurekubectl get hpaAutoscaler targets, current metric and replica counts. <unknown> in TARGETS means it cannot read the metric.
HPA not scaling
From inside a pod: test DNS and connectivity
kubectl exec -it payment-pod -- sh
# does the name resolve?
nslookup order-service
# does the Service answer?
curl -sv http://order-service:8080/actuator/healthRun these from the pod that is failing, not from your laptop: DNS settings and NetworkPolicies apply per pod. A name that does not resolve points at DNS; a name that resolves but hangs points at a NetworkPolicy or a Service with no endpoints.
Keep going
Sources: Kubernetes docs: kubectl quick reference ↗ · Kubernetes docs: Troubleshooting applications ↗ · Report a problem
Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.
Practise: diagnose the cluster
Read the terminal, watch the animation, pick the error and the first fix.