Skip to content

Consensus & replication · 5 of 6

Step 1 · The pattern

One leader, by lease

Only one worker may write to the storage at a time. A lock service gives one of them a lease: permission to be the leader for a short time. The leader keeps renewing it. The other worker stands by.

  • lease renewal
  • write
  • accepted

In short

Split brain & fencing tokens: how it works and how it breaks

A lease is a lock with a time limit: one node holds it, acts as the leader, and renews it before it runs out. If the leader dies, the lease expires and another node takes over. The weak point is that a node cannot tell when it has stopped. A long garbage-collection pause, a frozen VM or a slow network can hold a leader still for longer than its lease. When it resumes it still believes it leads, and it writes. Now two nodes act as leader, and the old one can overwrite newer data. No timeout fixes this, because the stall can happen after any check. A fencing token does. The lock service gives out a number that goes up with every grant. Every write carries it, and the storage refuses any write with a number lower than the highest it has seen.

How this comes up in interviews

"A worker holds a lease to be the only writer to some storage. It pauses for longer than the lease and then wakes up. What happens, and how do you keep the data safe?"

Then they ask

  1. Why does checking the lease against the local clock before each write not solve it?
  2. Where does the fencing token come from, and where must it be checked?
  3. The leader also calls other services or publishes messages. How do you protect those side effects from a stale leader?

A strong answer covers

  • A lease is a lock with a time limit. The holder acts as leader and renews it, and if it dies the lease expires and another node takes over.
  • A node cannot tell that it has stopped: a garbage-collection pause, a frozen VM or a slow network can outlast the lease.
  • On waking it still believes it leads and writes, so two nodes act as leader and the old one can overwrite newer data.
  • No timeout fixes this, because the stall can happen after any check and before the write lands.
  • A fencing token goes up with every grant, every write carries it, and the storage refuses any write with a number lower than the highest it has seen.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Engineering write-up

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.