Skip to content
All outages

Incident · Medium · DNS/HTTP · Infra

Fixing DNS didn't fix it

  • SEV1
  • 07:30 PST
  • INTERNAL NETWORK
90s

The situation

An automated scaling job runs on your internal network at 07:30. Soon internal DNS lookups fail, the console is unreachable and your monitoring goes dark. The team moves internal DNS away from the congested path and lookups recover. Everything else stays broken.

The evidence

Internal network: DNS errors vs connection attemptsillustrative
  • DNS errors
  • connection attempts
Logs and signalsReconstructed
07:30automated scaling activity on internal network
07:31clients: connection surge to network devices
07:33internal DNS: errors
09:28DNS moved off congested path → DNS errors gone
09:30devices still congested · clients retrying without backing off

Your call

What is keeping the outage alive?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem