Skip to content

Consensus & replication · 1 of 6

Step 1 · The pattern

One node takes the writes

Every write goes to one node, the leader. It sends each change on to the followers, which keep copies. One writer means one order of changes, and the followers are ready to take over.

  • write
  • acknowledged: 'saved'
  • copy for a follower

In short

Leader & followers: how it works and how it breaks

The most common way to keep several copies of a database is to pick one node as the leader. Every write goes to the leader, which sends the change to the other nodes, the followers. If the leader fails, a follower is promoted and takes over. The detail that matters is when the leader tells the client 'saved'. With asynchronous replication it answers as soon as it has the write itself and ships it afterwards. That is fast, but if the leader crashes in between, the promoted follower never received the write, and the client was told it was safe. With synchronous replication the leader answers only after at least one follower confirms it has the write, so an acknowledged write always exists in two places. The price is one extra round trip on every write, and writes that stall when no follower can confirm.

How this comes up in interviews

"A client was told its write was saved, the leader crashed, and after failover the write is gone. How can that happen, and how would you prevent it?"

Then they ask

  1. With synchronous replication, the only follower that confirms writes goes down. What happens to writes, and how do you avoid it?
  2. A client sends a write and gets no answer before the leader crashes. What state is the write in, and what should the client do?
  3. The follower has confirmed a write. Can a read sent to that follower still miss it? Why?

A strong answer covers

  • All writes go to one leader, which gives one order of changes and sends each change to followers that can take over.
  • With asynchronous replication the leader answers once it has the write itself, so a crash before shipping loses a write the client believes is safe.
  • With synchronous replication the leader answers only after at least one follower confirms, so an acknowledged write exists in two places.
  • The price is one extra round trip on every write, and writes that stall when no follower can confirm.
  • The promise covers acknowledged writes only. A write with no answer is in an unknown state, so retry it with the same idempotency key.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.