Databases & caching · Production incident
After the failover, half the orders go missing
The database primary failed over in the middle of a deploy, and some confirmed orders are now missing. How should a failover work so that every write goes to one primary?
Thursday 14:26 · order-service, PostgreSQL primary and one replicaIllustrative scenario, not a real incident
Support: 212 customers have an order confirmation and no such order in their history
At 14:02, halfway through a rolling deploy of the order service, the PostgreSQL primary stopped answering. The failover manager promoted the replica at 14:03 and moved the DNS name for the writer to it.
The deploy paused itself on the error spike with six of twelve instances replaced, and is still paused. Errors were back to normal by 14:04. Since then, about half of all new orders cannot be found afterwards.
What the dashboards and logs show
- Promotion
- 14:03:10, replica promoted by the failover manager
- Deploy
- paused at 14:02, 6 of 12 instances on the new version
- Error rate
- normal since 14:04
- Missing orders
- 212 so far, all confirmed to the customer by email
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Orders are being written to two servers as you speak. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.