Skip to content
All outages

Incident · Hard · URL shortener · Databases

Twenty seconds for a connection

  • INTERMITTENT
  • TWO WEEKS
  • 24 DEPLOYS
90s

The situation

For two weeks your app times out in short bursts, then recovers within a minute. A trace of one failed request shows it waited almost 20 seconds just to be lent a database connection from the app's own pool. Database load looks normal. Traffic is normal. No release changed how you use the database. The waiting is spread across many endpoints, with no single slow query to blame. You speed up the worst queries anyway. Days later it happens again.

The evidence

Pool wait time vs database loadillustrative
  • wait for a connection
  • database load
Logs and signalsReconstructed
day 1errors: a flurry of context canceled
day 1trace: almost 20s waiting for a pooled connection
day 1slow-query fixes shipped: indexes, a materialised view
day 3timeouts again; no pattern in which requests are slow
day 31s lock timeout on every transaction → nothing trips
day 4timeouts again

Your call

No query is slow. What do you do next?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last reviewed: Engineering write-up