Incident · Easy · DNS/HTTP · Infra
The off switch that broke it
- SEV1
- 25 MIN
- ~28% OF TRAFFIC
90s
The situation
Roughly a quarter of HTTP traffic through your edge suddenly returns 500. It only hits customers on the older proxy who use the managed ruleset. Earlier today the team raised the request body buffer to 1 MB, then switched off an internal test tool.
The evidence
Logs and signalsReconstructed
t-40medge: body buffer raised to 1 MB (gradual rollout)t-1m global config: killswitch ON for internal WAF test rulet-0 config propagated fleet-wide in secondst+0 old proxy (lua): attempt to index a nil valuet+0 new proxy: no errors
Your call
Which change caused the 500s?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.