Use case · Fintech
Payment retry, double charge
The card is charged, the reply gets lost, the app retries. Watch one tap on Pay become two charges, then see an idempotency key turn the retry into a replay.
Used by apps like Stripe, Airbnb, Uber, PayPal and Shopify checkout
Happy path · break it · fix it
Payment retry, double charge: sequence diagram
Payment retry, double charge, shown as messages between Client (The customer's app or browser), API (The API gateway that receives the app's requests and passes them on), Payments (The payment service and its own database), Provider (The external card payment provider that charges the card).
Happy path: 8 steps.
- Client → API: POST /pay. The customer taps Pay once, and the app waits up to ten seconds for an answer.
- API → Payments: charge £40. The API passes the request to the payment service.
- Payments → Provider: charge card. The payment service asks the card provider to take £40 from the card.
- Provider → Payments: approved (reply). The provider charges the card once and says so.
- Payments: payment saved: paid. The payment service records the payment as paid in its database.
- Payments → API: 200 paid (reply). The good news starts its way back to the app.
- API → Client: 200 paid (reply). The answer arrives well inside the ten seconds, so the app stops waiting.
- Client: shows: paid. The customer sees a receipt: one tap, one charge.
Result: One tap, one charge, one receipt.
Break it: 10 steps.
- Client → API: POST /pay. The same tap on Pay, and the same ten-second wait for an answer.
- API → Payments: charge £40. The API passes the request to the payment service.
- Payments → Provider: charge card. The payment service asks the card provider to take £40 from the card.
- Provider → Payments: approved (reply). The provider charges the card. From this moment the money has moved.
- Payments → API: 200 paid (never arrives) (reply). The reply is lost on the way back. The app's timeout runs out and it cannot tell if it paid.
- Client → Payments: retry: POST /pay. So the app sends the request again, through the API, exactly as it did the first time.
- Payments: no key: looks new. Nothing in the retry ties it to the first attempt, so the service treats it as a new payment.
- Payments → Provider: charge card. It asks the provider to charge the same card for the same order a second time.
- Provider → Payments: approved (reply). To the provider these are two valid requests, so it charges the card again.
- Payments → Client: 200 paid (reply). This time the reply gets through. The app shows one payment; the card shows two.
Result: One tap on Pay, £40 taken twice.
Fix it: 10 steps.
- Client → API: POST /pay K7. The same tap. The app first makes up a random key, K7, for this one payment and sends it along.
- API → Payments: charge, K7. The API passes the request on with its key.
- Payments: insert K7: started. Before calling anyone, the service saves K7 as started. A retry now would get “in progress”.
- Payments → Provider: charge card. Only then does it ask the provider to take £40, with no database transaction held open.
- Provider → Payments: approved (reply). The provider charges the card once and says so.
- Payments: K7 = paid, 1 commit. The payment row, the key and the exact response are saved in one transaction: all or nothing.
- Payments → API: 200 paid (never arrives) (reply). The same reply is lost in the same place, and the app's timeout runs out as before.
- Client → Payments: retry, same key K7. The app retries through the API with the same key, since it is the same payment.
- Payments: K7 found: replay. The service finds K7 already finished, so it does no work and never calls the provider.
- Payments → Client: 200 paid (reply). It sends the stored response again. The app shows paid, and the card was charged once.
Result: Two requests, one charge: the retry got the first answer.
step 1 of 8
The customer taps Pay once, and the app waits up to ten seconds for an answer.
- request
- success
- waiting / retrying
The happy path, step by step
- The customer taps Pay and the app sends one payment request, then waits a limited time for the answer.
- The API passes the request to the payment service.
- The payment service asks the card provider to charge the card.
- The provider charges the card once and replies that the charge was approved.
- The payment service records the payment as paid in its own database.
- The response travels back through the API to the app before the app's timeout runs out.
- The customer sees a receipt for one charge.
The failure
What breaks and why
A timeout tells the caller that no answer came, not that nothing happened. If the reply is lost after the provider has charged the card, the app sees the same silence as if the request had never arrived, so it retries, or the customer taps Pay again. Without anything that marks the second request as the same payment, the payment service starts from scratch and the provider, which sees two valid requests, charges twice. This needs no traffic at all: one customer on a weak connection is enough, and every layer that retries by itself adds another chance.
Before / after
The fix
The client creates one random key per payment and sends it with every attempt. The payment service inserts the key into a table with a unique constraint before it calls the provider, and when the provider answers it saves the payment, the key's status and the exact response in one transaction. A retry with a finished key gets the stored response and no second call. A retry that arrives while the first attempt is still running gets “in progress” (409) and comes back later. If the service dies between the provider's answer and the commit, the retry must ask the provider what happened, or pass the same key on to it, before charging.
POST /pay HTTP/1.1
Content-Type: application/json
{"order": "A-1001", "amount": 4000}
-- server: runs for the first try AND the retry
INSERT INTO payments (order_id, amount, status)
VALUES ('A-1001', 4000, 'started');
-- call the provider: charge the card
UPDATE payments SET status = 'paid' WHERE id = :id;POST /pay HTTP/1.1
Idempotency-Key: 7c1e9f52-... (same on a retry)
BEGIN; -- 1. claim the key (unique column)
INSERT INTO idem_keys VALUES (:key, 'started'); -- exists? 3
INSERT INTO payments (...) VALUES (...);
COMMIT;
-- 2. call the provider, no transaction open
BEGIN; -- result and response together
UPDATE payments SET status = 'paid' WHERE id = :id;
UPDATE idem_keys SET status = 'done', response = :r
WHERE key = :key;
COMMIT;
-- 3. key exists: done -> stored reply, started -> 409Boss fight
A related pattern in a real outage
Related pattern
This card is not an outage of this exact flow. It shows the same mechanism failing in another product.
A related pattern in another product, not this exact flow: customers' cards are charged over and over for account top-ups, and the payments team says every charge is correct.
Fight the related boss90 seconds · the company is revealed after you answer
From their own write-ups
How real companies approach it
Stripe describes accepting an Idempotency-Key header on the POST endpoints of its API: when a request succeeded but the client never got the result, a retry with the same key is answered with the saved result of the first. It also says its Ruby library retries by itself with a key, growing waits and jitter.
Stripe: Designing robust and predictable APIs with idempotency (2017) ↗Airbnb describes Orpheus, an idempotency library used across its payment services. It splits a request into three phases: record the request, call the outside service, record the response. The database work of the first and last phase is one transaction each, and no network call happens inside either.
Airbnb Tech Blog: Avoiding double payments in a distributed payments system (2019) ↗In the same post Airbnb says each request takes a row lock on its key as a lease that expires, set longer than the timeout of the outside call, and that key data is read and written on the primary database only, since a lagging replica could hide a stored response and let a payment run twice.
Airbnb Tech Blog: Avoiding double payments in a distributed payments system (2019) ↗
How this comes up in interviews
"A customer taps Pay, the app times out and tries again. How do you make sure the card is charged only once?"
Then they ask
- A retry arrives while the first attempt is still calling the provider. What does the payment service do?
- The service dies after the provider charged the card but before it committed the result. How does the retry stay safe?
- Who creates the key, and why not derive it on the server from the contents of the request?
A strong answer covers
- A timeout tells the caller that no answer came, not that nothing happened.
- The client creates one random key per payment and sends it with every attempt.
- The service inserts the key under a unique constraint before it calls the provider.
- It saves the payment, the key's status and the exact response in one transaction. A finished key gets the stored response.
- A retry during the first attempt gets in progress (409). After a crash, ask the provider or pass the same key on before charging.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
Your turn
What would you do?
Support reports customers charged twice for one order. The logs show the provider approving a charge, the app timing out after 10 seconds on a weak connection, the app retrying, and a second approval. Which change removes the cause?
Related
Payment retry, double charge questions
- What is an idempotency key?
- A unique value, usually a random UUID, that a client creates for one operation and sends with every attempt of it, often in an Idempotency-Key header. The server stores the key with the outcome, so a repeat of the same request returns the first result and the work is done once.
- Why can a payment timeout lead to a double charge?
- A timeout only means the answer did not arrive. The charge may have gone through with the reply lost afterwards. If the client retries and the server cannot tell that the retry is the same payment, it charges again. The key gives the server a way to tell.
- What should an API return when a retry arrives while the first request is still running?
- It should not start the work a second time. The IETF draft for the Idempotency-Key header suggests 409 Conflict for a retry whose original is still being processed, and 422 when a key is reused with a different request body. The client waits and retries with the same key.
Sources: Stripe: Designing robust and predictable APIs with idempotency (2017) ↗ · Airbnb Tech Blog: Avoiding double payments in a distributed payments system (2019) ↗ · IETF Internet-Draft: The Idempotency-Key HTTP Header Field (work in progress) ↗ · Twilio: Billing incident post-mortem (July 2013), the source of the related Break card ↗ · Report a problem
A typical design, informed by public write-ups, not any one company's exact system. Diagrams are our own. Numbers in the simulation are illustrative. All wording is original and paraphrases the linked sources.