Skip to content
Interview rounds

Concurrency & payments · Production incident

Cancelled, and still charged

An order fails at its last step, and the step that should undo the payment fails too. How do you keep the system consistent?

BackendStaff14 min
Interview-style scenarioLast reviewed: Checked against official documentation

Tuesday 10:40 · order-serviceIllustrative scenario, not a real incident

Finance: 64 customers were charged for orders that were cancelled yesterday evening

The order service runs each order in three steps, each a call to another service: reserve the stock, take the payment, book the shipping. When a step fails, it undoes the earlier steps in reverse order.

Yesterday from 19:10 to 19:50 the shipping partner's API was down, and the orders placed in that window were cancelled. Of those, 64 still show a captured payment with no refund, and 19 still hold their stock.

What the dashboards and logs show

Orders cancelled in the window
1,930, each with the status CANCELLED
Cancelled and not refunded
64
Cancelled with stock still reserved
19
Payment service
p99 of 9 seconds from 19:15 to 19:40, 300 ms on a normal evening

Clarify · optional · scored under "Hypothesis before change"

Ask the interviewer

0 of 3 asked

Each question reveals a fact. What you ask before you act is part of your score.

First move · scored under "Mitigate before root cause"

The outage is over, and 64 customers are still out of pocket. What do you do first?

Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.