Skip to content
All outages

Incident · Medium · Infra

Overloaded, yet idle

  • SEV1
  • FIRST WORKDAY OF THE YEAR
90s

The situation

It's the first Monday after the holidays. Millions of clients reconnect at once. Latency explodes and requests time out. Weirdly, your web tier looks… relaxed.

The evidence

Web tier CPU vs p99 latencyillustrative
  • CPU %
  • p99 latency
Logs and signalsReconstructed
14:02packet loss between VPCs ↑
14:05autoscaler: CPU low → scaling DOWN web tier
14:20autoscaler: errors high → scaling UP +1,200
14:21provisioner: EMFILE: too many open files
14:22dashboards: partial data

Your call

What's really going on?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last reviewed: Official postmortem