Databases & caching · Production incident
Slow at peak, with no deadlock in sight
At peak, some writes wait many seconds in PostgreSQL and no deadlock is reported. How do you find what they wait for and stop it?
Wednesday 12:25 · ledger-serviceIllustrative scenario, not a real incident
ledger-service: capture p99 reaches the 10 s gateway timeout at lunch. deadlocks counter: 0
The ledger service records each payment capture in PostgreSQL. For the third day running, about one capture in eight fails with a 504 between 12:10 and 12:50.
Outside that window the same requests take about 60 ms. A release went out on Monday morning. The database has spare CPU and has reported no errors.
What the dashboards and logs show
- Capture latency at peak
- p50 60 ms, p99 10 s (the gateway timeout)
- Failed captures at peak
- about 12%, all 504 after 10 s
- Database
- CPU 30%, deadlocks 0, no errors logged
- Connection pools
- 31 of 40 in use on the busiest instance
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
It is 12:25 and captures are failing. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.