Skip to content
All outages

Incident · Medium · DNS/HTTP · Databases

The network that keeps blinking

  • SEV1
  • 11:20 UTC
  • GLOBAL
90s

The situation

Your global proxy fleet starts returning 5xx on a huge share of traffic. Then it recovers. Then it fails again. Your status page, hosted somewhere else entirely, is also down.

The evidence

5xx rate, all regionsillustrative
  • 5xx %
Logs and signalsReconstructed
10:59bot-features.file size=1.0x
11:05db: ClickHouse permissions change applied
11:20bot-features.file size=2.0x
11:20proxy panic: called unwrap() on Err (features > 200)
11:25bot-features.file size=1.0x
11:30bot-features.file size=2.0x
-- file regenerated every 5 min

Your call

What's actually taking the network down?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last reviewed: Official postmortem