Skip to content
All outages

Incident · Medium · Resilience

Every restart freezes again

  • SEV1
  • 16:20 UTC
  • SHARED GATEWAY
90s

The situation

Your mail service sits behind a connection gateway it shares with the company's chat product. Overnight, chat failed over onto this gateway, a routine move, while a slice of gateway servers was already out for repair. At the evening peak, servers stop responding one after another. Each batch you restart takes traffic and freezes again. Someone says: add more servers.

The evidence

Inbound traffic vs unresponsive serversillustrative
  • inbound requests
  • % servers unresponsive
Logs and signalsReconstructed
01:07failover: chat traffic moved onto the primary gateway
01:07gateway capacity in service: 86% (repair ticket open)
16:03anomaly monitor: connection failures rising
16:17auto-redirect: traffic moved off unhealthy endpoints
19:20inbound traffic: normal range; servers unavailable: rising
23:55restarted servers healthy, then unresponsive once traffic arrives
00:30connection handler: CPU 100%, lock wait

Your call

What is taking the gateway down?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last reviewed: Official postmortem