Practice
SEV-1 simulationCaching and state · Real outage · Reconstructed data · 15–20 minutes
Incident #005 — The cache that never refills
Customer impact
Clients are failing to boot. The membership database is saturated and the cache in front of it is not refilling.
| Signal | Current value |
|---|---|
| Client boot errors | high, rising |
| Cache hit rate | ≈70%, flat |
| Membership DB load | saturated |
| Shard query timeouts | most queries |
| Memcached nodes | all up |
| Consul upgrade | 25% step running |
Your task
You are on call for the API tier. Work out why the cache is not recovering, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.
06:20Impact continuing · +0 minabout 0 thousand failed client boots lost since you joined
Investigate · each first look costs 2 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time