Skip to content

Cheat sheet

Essential kubectl troubleshooting commands

Fourteen commands cover most production investigations. Each row says what the command is for and links to the problem it most often uncovers. Add -n <namespace> to any of them.

Pods: what is running, and why not

  • kubectl get pods

    List pods with READY, STATUS and RESTARTS. The first look at any problem.

    CrashLoopBackOff
  • kubectl describe pod POD

    State, last state with exit code, probe settings and the events for one pod.

    Pending pod
  • kubectl logs POD

    Output of the current container.

    Liveness probe failure
  • kubectl logs POD --previous

    Output of the container instance that ended before the last restart. This is where a crash explains itself.

    CrashLoopBackOff
  • kubectl get events --sort-by=.metadata.creationTimestamp

    Recent events across the namespace in time order: scheduling, pulls, probes, kills.

    ImagePullBackOff

Resources: memory and CPU

  • kubectl top pods

    Current CPU and memory per pod. Needs metrics-server. Add --containers for each container.

    OOMKilled
  • kubectl top nodes

    Current CPU and memory per node, to see whether the machines are full.

    CPU throttling
  • kubectl get nodes

    Node health: Ready, NotReady or SchedulingDisabled.

    Node NotReady

Traffic: Services and their endpoints

  • kubectl get svc

    List Services with their cluster IP and ports.

    Service has no endpoints
  • kubectl get endpointslices

    The pod addresses a Service actually sends traffic to. Empty means no pod is selected or none is Ready.

    Readiness probe failure

Releases and scaling

  • kubectl get deployments

    READY, UP-TO-DATE and AVAILABLE counts for each Deployment.

    Deployment rollout failure
  • kubectl rollout status deploy/APP

    Follow a rollout until it finishes or reports that it has stalled.

    Deployment rollout failure
  • kubectl rollout undo deploy/APP

    Return to the previous revision. The fastest safe move when a release is the cause.

    Deployment rollout failure
  • kubectl get hpa

    Autoscaler targets, current metric and replica counts. <unknown> in TARGETS means it cannot read the metric.

    HPA not scaling

From inside a pod: test DNS and connectivity

Example
kubectl exec -it payment-pod -- sh

# does the name resolve?
nslookup order-service

# does the Service answer?
curl -sv http://order-service:8080/actuator/health

Run these from the pod that is failing, not from your laptop: DNS settings and NetworkPolicies apply per pod. A name that does not resolve points at DNS; a name that resolves but hangs points at a NetworkPolicy or a Service with no endpoints.

Sources: Kubernetes docs: kubectl quick reference ↗ · Kubernetes docs: Troubleshooting applications ↗ · Report a problem

Terminal output is illustrative. All wording is original and paraphrases the linked Kubernetes documentation.

Practise: diagnose the cluster

Read the terminal, watch the animation, pick the error and the first fix.

Play