Databases & caching · Production incident
One entry expires, the database goes down
A burst of cache misses overloads the database at the busiest time of day. How do you stop a missing cache entry from doing that?
Thursday 12:31 · catalogue-serviceIllustrative scenario, not a real incident
catalogue-service: 5xx rate 41%, database CPU 100% for the last three minutes
The shop's front page shows a top sellers listing. The catalogue service keeps it in Redis and reads PostgreSQL when it has to. Lunchtime is the daily peak, at about 800 requests a second.
At 12:28 the database went to full CPU and its connections ran out. Nothing was deployed today, and traffic is no higher than on any other Thursday.
What the dashboards and logs show
- Database CPU
- 100% since 12:28, about 15% a minute earlier
- Database connections
- 200 of 200 in use
- catalogue-service
- 41% of responses are 503 or 504
- Redis
- up, replies in under a millisecond
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
The site is failing at its busiest hour. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.