Skip to content

Redis / caching · 7 of 9

Step 1 · The pattern

A primary, a replica, a referee

App servers send commands to the primary. It streams every change to a replica. Sentinel watches both and always knows which one is the primary.

  • command
  • answered
  • replication

In short

Failover & stale connections: how it works and how it breaks

High availability for Redis has two halves. The server half is Sentinel: it watches the primary, and when the primary stops answering it promotes a replica and records the new address. The client half is every application noticing the change and reconnecting. If the old primary's process crashes, its connections are reset and clients find out at once. If the machine freezes or drops off the network, nothing comes back at all. A client with no socket timeout will sit on those dead connections indefinitely, while a healthy new primary waits unused. The fix is on the client. Set connect and read timeouts. On any timeout or error, ask Sentinel for the current primary again, check its role, and replace every pooled connection that points at the old address. Test it by cutting the network to a primary, not only by killing the process.

How this comes up in interviews

"Your Redis primary's machine drops off the network. Sentinel promotes a replica within seconds, yet the application stays down. Why?"

Then they ask

  1. How does a client find the new primary, and how does it know the answer is correct?
  2. Why does kill -9 on the primary not prove that failover works?
  3. Can a write that Redis acknowledged before the failover be lost? Why?

A strong answer covers

  • Failover has a server half, where Sentinel promotes a replica, and a client half, where every application must notice and reconnect.
  • A crashed process resets its connections. A frozen or unreachable machine sends nothing back at all.
  • Without a socket timeout, clients wait on dead connections indefinitely while the new primary sits unused.
  • On any timeout or error: ask Sentinel for the current primary, check its role, and replace every pooled connection to the old address.
  • Test by cutting the network to the primary, not only by killing the process.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.