Skip to content
All outages

Incident · Medium · DNS/HTTP · Infra

Fine until you touch it

  • SEV1
  • CHRISTMAS EVE
  • LOAD BALANCERS
90s

The situation

It is Christmas Eve. A few customers report their load balancers misbehaving. Most are fine. Over the next hours the number creeps up, peaking near 7%. The pattern is strange: a load balancer works normally until it scales or someone edits its settings, and then it breaks.

The evidence

Load balancers misconfiguredillustrative
  • affected %
Logs and signalsReconstructed
12:24maintenance job run against production state store
12:25control plane: load balancer state records missing
13:10lb-4417 scaled up → rebuilt from state → misconfigured
13:11lb-0093 untouched → serving normally
20:00first state restore attempt failed

Your call

What explains the pattern?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem