Skip to content

Consumer apps at scale · 9 of 13

Step 1 · The pattern

Read from the cache, check the value

Every page needs a configuration value. Servers read it from the cache, 10 times a second, and a checker in the client confirms it looks valid. The database holds the source of truth and is barely touched. (All numbers are illustrative.)

  • read
  • valid value

In short

The self-repair loop: how it works and how it breaks

Automatic repair is a good idea with a dangerous edge. A common example: every server reads a configuration value from a cache, and the client checks it; if the value looks invalid, the client fetches a fresh copy from the database. That fixes a corrupt cache entry. When the database itself holds a value the checker rejects, every server repairs the same key at the same moment, all against the database. The database is overwhelmed, its errors are mistaken for invalid values, and clients delete their cached copy, which sends even more requests. The loop can outlast the bad value. A large social network was down for about two and a half hours this way. Bound every automatic repair: one repair per key at a time, never treat an error as bad data, and when the source of truth agrees, stop and alert a person.

What this simulation simplifies

  • One cache entry and one checker stand for every server's copy of the value. In the real outage, each client ran its own check.
  • The database answers two queries at a time and refuses beyond eight. A real database slows down first, then times out.
  • The bad value is reverted after five seconds. The real incident lasted about two and a half hours.

How this comes up in interviews

"Your clients verify a cached config value and fetch it from the database when it looks invalid. What can go wrong, and how do you design it safely?"

Then they ask

  1. The database itself now holds a value the checker rejects. What happens?
  2. How should a client behave when the database returns an error?
  3. The value has been fixed but the database is still overloaded. How do you recover?

A strong answer covers

  • If the source of truth holds the rejected value, every client repairs at once and floods the database.
  • If errors are treated as invalid values, clients delete cache entries and the loop feeds itself after the fix.
  • Bound repairs: one lease per key, a fleet-wide rate limit and jittered back-off.
  • On errors keep the last good value; when the source agrees with the rejected value, stop and alert a person.
  • Have a kill switch for the repair logic, tested for the case where the database is unreachable.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Official postmortem

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.