Databases & caching · Production incident
No connections left, and an idle database
Requests time out waiting for a database connection while the database is almost idle. How do you stop the pool from running out?
Tuesday 14:12 · order-serviceIllustrative scenario, not a real incident
order-service: 9 in 10 orders fail, each after almost exactly 30 seconds
The order service is a Spring Boot application on four instances. Each has a HikariCP pool at its default size and talks to one PostgreSQL database. It takes about 12 orders a second per instance.
Since 14:05 most orders hang and then fail. The few that succeed take about nine seconds. Nothing was deployed today, and the database dashboards look calm.
What the dashboards and logs show
- Failed orders
- about 90%, each after 30 s, with an SQLException
- Database CPU
- 4%
- Slowest query in the last hour
- 14 ms
- Deploys
- none today
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Nine in ten orders are failing. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.