Databases & caching · Production incident
Transfers that abort each other
Under load, some transactions fail with a deadlock error from PostgreSQL. How do you stop them deadlocking?
Wednesday 10:40 · wallet-serviceIllustrative scenario, not a real incident
wallet-service: 1.4% of transfers return 500, with ERROR: deadlock detected in the log
The wallet service moves money between two accounts in one PostgreSQL transaction. It handles about 600 transfers a second, and until this week the error rate was close to zero.
Since Tuesday afternoon a steady trickle of transfers fails. The p99 latency of a transfer has gone from 40 ms to 1.1 seconds. The database is not short of CPU or connections.
What the dashboards and logs show
- Failed transfers
- 1.4%, all with the same error
- Transfer latency
- p50 12 ms as before, p99 1.1 s, up from 40 ms
- Database CPU
- 22%, no change
- Balances
- reconcile to the penny: no money lost or created
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Transfers are failing for real users while you look into it. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.