Incident · Easy · URL shortener · Databases
Same customers, same sick server
- SEV2
- ~7 HOURS OVERNIGHT
- ONE HOST
90s
The situation
You run customers' small programs on a fleet of servers. To keep them warm, the router sends each program's requests to the same server every time. Just after midnight a burst of parallel edits hits one program. From then on a fixed group of customers see their requests hang, while most others are fine. Your outside health check stays green all night. Nobody is paged. A customer reports it nearly seven hours later.
The evidence
- stuck connections
- pool size
Logs and signalsReconstructed
00:15host-3: many parallel edits to one program00:15host-3 db: deadlock; queries never return00:16host-3 pool: 0 free; requests pending, none rejected00:20external health check → routed to host-1: 200 OK03:00external health check → routed to host-2: 200 OK07:04user report → host-3 terminated by hand
Your call
Why did this last seven hours without an alert?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.