Databases & caching · Concept deep-dive
A cache on four machines loses one
How do you design a cache that runs on many machines, so that losing or adding a machine does not flood the database?
Concept deep-dive · a cache spread over several machinesIllustrative scenario, not a real incident
Review: last month one of four cache nodes failed, and the database was saturated for 20 minutes
A catalogue service keeps product data on four standalone Redis nodes. Its client library picks the node for a key with hash(key) mod 4, and on a miss the service reads PostgreSQL and stores the result.
Last month one node failed. The clients dropped it from their list and carried on with three. A fifth node is planned for next week, because memory is filling up.
What the dashboards and logs show
- Reads
- 200,000 a second, 96% answered from the cache
- Database
- serves 8,000 reads a second normally, 80,000 at most
- Cache nodes
- 4 × 32 GB, each about 70% full
- Last month
- one node lost, database saturated for 20 minutes
5 questions on one topic, each at least as hard as the last. Answer each before the next appears. The last one plays out.
Question 1 of 5 · Mid · scored under "Hypothesis before change"
The failed node held a quarter of the keys. Why was the database saturated, and not a little busier?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.