Skip to content
All outages

Incident · Easy · URL shortener · Databases

Same customers, same sick server

  • SEV2
  • ~7 HOURS OVERNIGHT
  • ONE HOST
90s

The situation

You run customers' small programs on a fleet of servers. To keep them warm, the router sends each program's requests to the same server every time. Just after midnight a burst of parallel edits hits one program. From then on a fixed group of customers see their requests hang, while most others are fine. Your outside health check stays green all night. Nobody is paged. A customer reports it nearly seven hours later.

The evidence

Database connections on one serverillustrative
  • stuck connections
  • pool size
Logs and signalsReconstructed
00:15host-3: many parallel edits to one program
00:15host-3 db: deadlock; queries never return
00:16host-3 pool: 0 free; requests pending, none rejected
00:20external health check → routed to host-1: 200 OK
03:00external health check → routed to host-2: 200 OK
07:04user report → host-3 terminated by hand

Your call

Why did this last seven hours without an alert?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem