Incident · Medium · Infra
The label that disappeared
- SEV1
- PI DAY
- >5 HOURS
90s
The situation
You upgrade a production Kubernetes cluster to version 1.24. The upgrade completes cleanly. Then pods across the cluster lose the ability to talk to each other. The first error anyone spots is an admission webhook timing out.
The evidence
Logs and signalsReconstructed
19:00control plane upgraded to 1.24 OK19:05policy webhook: timeout19:06pod → pod: no route19:20route reflectors selected: 0 nodes-- selector: node-role.kubernetes.io/master
Your call
What broke pod networking?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.