Incident · Medium · DNS/HTTP · Infra
Fine until you touch it
- SEV1
- CHRISTMAS EVE
- LOAD BALANCERS
90s
The situation
It is Christmas Eve. A few customers report their load balancers misbehaving. Most are fine. Over the next hours the number creeps up, peaking near 7%. The pattern is strange: a load balancer works normally until it scales or someone edits its settings, and then it breaks.
The evidence
- affected %
Logs and signalsReconstructed
12:24maintenance job run against production state store12:25control plane: load balancer state records missing13:10lb-4417 scaled up → rebuilt from state → misconfigured13:11lb-0093 untouched → serving normally20:00first state restore attempt failed
Your call
What explains the pattern?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.