Concurrency & payments · Production incident
Paid, and still pending
The provider took the payment, and the order still says it is waiting for it. How do you make sure a paid order is marked as paid?
Thursday 14:10 · order-serviceIllustrative scenario, not a real incident
Support: 212 customers say they have paid, and their orders still show Awaiting payment
At checkout the customer pays on the payment provider's hosted page. The provider then calls POST /callbacks/payment on the order service, which moves the order from PENDING to PAID.
All 212 orders were placed between 13:05 and 13:20 today. The customers have a confirmation from their bank, and some of them have placed the same order again.
What the dashboards and logs show
- Orders in PENDING
- 212 from 13:05 to 13:20, a normal quarter of an hour has under 5
- Provider dashboard
- all 212 payments shown as successful
- Order service
- healthy now, checkout success rate normal since 13:20
- Release
- a gateway routing change went out at 13:05 and was rolled back at 13:20
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Customers have paid and cannot see their orders move. Some are paying again. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.