Skip to content
All outages

Incident · Easy · DNS/HTTP · Infra

The off switch that broke it

  • SEV1
  • 25 MIN
  • ~28% OF TRAFFIC
90s

The situation

Roughly a quarter of HTTP traffic through your edge suddenly returns 500. It only hits customers on the older proxy who use the managed ruleset. Earlier today the team raised the request body buffer to 1 MB, then switched off an internal test tool.

The evidence

Logs and signalsReconstructed
t-40medge: body buffer raised to 1 MB (gradual rollout)
t-1m global config: killswitch ON for internal WAF test rule
t-0 config propagated fleet-wide in seconds
t+0 old proxy (lua): attempt to index a nil value
t+0 new proxy: no errors

Your call

Which change caused the 500s?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last reviewed: Official postmortem