Skip to content
Interview rounds

Databases & caching · Production incident

After the failover, half the orders go missing

The database primary failed over in the middle of a deploy, and some confirmed orders are now missing. How should a failover work so that every write goes to one primary?

Platform-SRESenior14 min
Interview-style scenarioLast reviewed: Checked against official documentation

Thursday 14:26 · order-service, PostgreSQL primary and one replicaIllustrative scenario, not a real incident

Support: 212 customers have an order confirmation and no such order in their history

At 14:02, halfway through a rolling deploy of the order service, the PostgreSQL primary stopped answering. The failover manager promoted the replica at 14:03 and moved the DNS name for the writer to it.

The deploy paused itself on the error spike with six of twelve instances replaced, and is still paused. Errors were back to normal by 14:04. Since then, about half of all new orders cannot be found afterwards.

What the dashboards and logs show

Promotion
14:03:10, replica promoted by the failover manager
Deploy
paused at 14:02, 6 of 12 instances on the new version
Error rate
normal since 14:04
Missing orders
212 so far, all confirmed to the customer by email

Clarify · optional · scored under "Hypothesis before change"

Ask the interviewer

0 of 3 asked

Each question reveals a fact. What you ask before you act is part of your score.

First move · scored under "Mitigate before root cause"

Orders are being written to two servers as you speak. What do you do first?

Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.