Skip to content
Interview rounds

Kubernetes & deploys · Production incident

The failover that half worked

The primary region fails, traffic is moved to the second region, and things go wrong there. How should a failover between regions be designed?

Platform-SRESenior14 min
Interview-style scenarioLast reviewed: Checked against official documentation

Monday 09:25 · platform on-callIllustrative scenario, not a real incident

Region A is unreachable. Failover to region B began at 09:12, and 46% of requests are failing

The shop runs active-passive. Region A serves all traffic and holds the primary database. Region B has a standby database and a copy of every service.

Region A dropped off the network at 09:10. On-call followed the runbook at 09:12: promote the database in region B, then point the shop's DNS record at region B.

Thirteen minutes later about half of all requests still fail. Support also reports customers whose orders from the minutes before 09:10 are not in their accounts.

What the dashboards and logs show

Failed requests
46%, a mix of timeouts and 503
Requests still sent to region A's address
about 35% of the total
Region B checkout
CPU at 97%, pods restarting
Order rate before 09:10
about 12 a second
Last failover rehearsal
14 months ago, at night

Clarify · optional · scored under "Hypothesis before change"

Ask the interviewer

0 of 3 asked

Each question reveals a fact. What you ask before you act is part of your score.

First move · scored under "Mitigate before root cause"

Region A is still down and half of all requests are failing. What do you do first?

Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.