Kubernetes & deploys · Production incident
The failover that half worked
The primary region fails, traffic is moved to the second region, and things go wrong there. How should a failover between regions be designed?
Monday 09:25 · platform on-callIllustrative scenario, not a real incident
Region A is unreachable. Failover to region B began at 09:12, and 46% of requests are failing
The shop runs active-passive. Region A serves all traffic and holds the primary database. Region B has a standby database and a copy of every service.
Region A dropped off the network at 09:10. On-call followed the runbook at 09:12: promote the database in region B, then point the shop's DNS record at region B.
Thirteen minutes later about half of all requests still fail. Support also reports customers whose orders from the minutes before 09:10 are not in their accounts.
What the dashboards and logs show
- Failed requests
- 46%, a mix of timeouts and 503
- Requests still sent to region A's address
- about 35% of the total
- Region B checkout
- CPU at 97%, pods restarting
- Order rate before 09:10
- about 12 a second
- Last failover rehearsal
- 14 months ago, at night
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Region A is still down and half of all requests are failing. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.