Databases & caching · Production incident
One cache node is saturated, five are idle
One node of a Redis Cluster is saturated while the others are idle, and reads through it are slow. How do you bring latency back down?
Wednesday 19:20 · product-serviceIllustrative scenario, not a real incident
product-service: p99 above 800 ms for about one product page in six. Node redis-3 CPU 100%
A marketplace's product service keeps each product's details in a Redis Cluster of six primaries and reads PostgreSQL on a miss. Evenings are busy, at about 450,000 cache reads a second.
Since 19:15 one node has been at full CPU while the other five sit near 40%. Products that are slow stay slow, and the rest of the site is as fast as usual. Nothing was deployed today.
What the dashboards and logs show
- redis-3
- CPU 100%, about 150,000 reads a second
- Other five nodes
- CPU near 40%, about 60,000 reads a second each
- Memory
- about 55% used on every node, no evictions
- Cache hit ratio
- 99%, unchanged
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
A sixth of product pages are slow at the evening peak. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.