Incident · Medium · Resilience
Every restart freezes again
- SEV1
- 16:20 UTC
- SHARED GATEWAY
90s
The situation
Your mail service sits behind a connection gateway it shares with the company's chat product. Overnight, chat failed over onto this gateway, a routine move, while a slice of gateway servers was already out for repair. At the evening peak, servers stop responding one after another. Each batch you restart takes traffic and freezes again. Someone says: add more servers.
The evidence
- inbound requests
- % servers unresponsive
Logs and signalsReconstructed
01:07failover: chat traffic moved onto the primary gateway01:07gateway capacity in service: 86% (repair ticket open)16:03anomaly monitor: connection failures rising16:17auto-redirect: traffic moved off unhealthy endpoints19:20inbound traffic: normal range; servers unavailable: rising23:55restarted servers healthy, then unresponsive once traffic arrives00:30connection handler: CPU 100%, lock wait
Your call
What is taking the gateway down?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.