Skip to content
Interview rounds

Databases & caching · Production incident

One cache node is saturated, five are idle

One node of a Redis Cluster is saturated while the others are idle, and reads through it are slow. How do you bring latency back down?

BackendSenior12 min
Interview-style scenarioLast reviewed: Checked against official documentation

Wednesday 19:20 · product-serviceIllustrative scenario, not a real incident

product-service: p99 above 800 ms for about one product page in six. Node redis-3 CPU 100%

A marketplace's product service keeps each product's details in a Redis Cluster of six primaries and reads PostgreSQL on a miss. Evenings are busy, at about 450,000 cache reads a second.

Since 19:15 one node has been at full CPU while the other five sit near 40%. Products that are slow stay slow, and the rest of the site is as fast as usual. Nothing was deployed today.

What the dashboards and logs show

redis-3
CPU 100%, about 150,000 reads a second
Other five nodes
CPU near 40%, about 60,000 reads a second each
Memory
about 55% used on every node, no evictions
Cache hit ratio
99%, unchanged

Clarify · optional · scored under "Hypothesis before change"

Ask the interviewer

0 of 3 asked

Each question reveals a fact. What you ask before you act is part of your score.

First move · scored under "Mitigate before root cause"

A sixth of product pages are slow at the evening peak. What do you do first?

Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.