Incident · Hard · URL shortener · Databases
Twenty seconds for a connection
- INTERMITTENT
- TWO WEEKS
- 24 DEPLOYS
90s
The situation
For two weeks your app times out in short bursts, then recovers within a minute. A trace of one failed request shows it waited almost 20 seconds just to be lent a database connection from the app's own pool. Database load looks normal. Traffic is normal. No release changed how you use the database. The waiting is spread across many endpoints, with no single slow query to blame. You speed up the worst queries anyway. Days later it happens again.
The evidence
- wait for a connection
- database load
Logs and signalsReconstructed
day 1errors: a flurry of context canceledday 1trace: almost 20s waiting for a pooled connectionday 1slow-query fixes shipped: indexes, a materialised viewday 3timeouts again; no pattern in which requests are slowday 31s lock timeout on every transaction → nothing tripsday 4timeouts again
Your call
No query is slow. What do you do next?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.