Skip to content
Practice

SEV-1 simulationCaching and state · Real outage · Reconstructed data · 15–20 minutes

Incident #005 — The cache that never refills

Customer impact

Clients are failing to boot. The membership database is saturated and the cache in front of it is not refilling.

SignalCurrent value
Client boot errorshigh, rising
Cache hit rate≈70%, flat
Membership DB loadsaturated
Shard query timeoutsmost queries
Memcached nodesall up
Consul upgrade25% step running

Your task

You are on call for the API tier. Work out why the cache is not recovering, using the service map, metrics, logs, traces and recent changes. Stop the impact first, then name the cause and the permanent fix.

06:20Impact continuing · +0 minabout 0 thousand failed client boots lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

Users cannot load the app, and the cache is not coming back. What do you do first?