Skip to content
All outages

Incident · Medium · Infra

The label that disappeared

  • SEV1
  • PI DAY
  • >5 HOURS
90s

The situation

You upgrade a production Kubernetes cluster to version 1.24. The upgrade completes cleanly. Then pods across the cluster lose the ability to talk to each other. The first error anyone spots is an admission webhook timing out.

The evidence

Logs and signalsReconstructed
19:00control plane upgraded to 1.24 OK
19:05policy webhook: timeout
19:06pod → pod: no route
19:20route reflectors selected: 0 nodes
-- selector: node-role.kubernetes.io/master

Your call

What broke pod networking?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem