Skip to content

Consensus & replication · 6 of 6

Step 1 · The pattern

A primary, a replica, a watcher

App servers write to the primary in site A. It copies every change to a replica in site B. A failover tool keeps probing the primary and can promote the replica if the primary stops answering.

  • write
  • saved
  • copy to replica / probe

In short

Failover gone wrong: how it works and how it breaks

Automatic failover promotes a replica when the primary stops answering. That makes it a production change made by a program, in a hurry, on incomplete information. From outside, a dead primary and a short network cut look the same: silence. If the tool promotes at once during a cut, the old primary is still alive, holds writes the replica never received, and may accept more. The new primary accepts writes too. When the link returns there are two histories, and nothing can merge them safely: picking one side discards the other's writes or leaves people to reconcile them by hand. A safer failover waits a grace period longer than an ordinary blip, fences the old primary so it cannot take writes, and promotes only a replica that has every change. The cost is a short write pause. Rehearse it: a failover path that never runs is untested code.

How this comes up in interviews

"Your primary database stops answering health probes. Design the automatic failover: when do you promote the replica, and what must be true before you do?"

Then they ask

  1. A colleague wants to shorten the wait before promotion to recover faster from crashes. What is the risk?
  2. The old primary is cut off by a network partition, so you cannot tell it to stop. How do you fence it?
  3. Both sides took writes during the split. What do you do next, and why not fail back to the old primary?

A strong answer covers

  • From outside, a dead primary and a short network cut look the same: silence.
  • Promoting at once during a cut leaves the old primary alive with writes the replica never received, while the new primary accepts writes too.
  • The result is two histories that nothing can merge safely: one side's writes are discarded or reconciled by hand.
  • A safer failover waits a grace period longer than an ordinary blip, fences the old primary, and promotes only a replica that has every change.
  • The cost is a short write pause. Rehearse the path, because a failover that never runs is untested code.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Official postmortem

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.