Incident · Hard · Redis / caching
Full every ten minutes
- SLO BURN
- EVERY FEW MINUTES
- FOR MONTHS
90s
The situation
A large Redis instance used purely as an LRU cache has a strange heartbeat. Every several minutes, latency spikes and throughput drops for a few seconds, then everything is fine again. Client traffic is steady. On the memory graph, usage climbs to maxmemory, flat-tops, then suddenly falls by hundreds of megabytes. No key in the dataset is anywhere near that big.
The evidence
- memory
- requests over 1s
Logs and signalsReconstructed
t+0m memory below maxmemory · evictions: 0t+9m memory = maxmemory · evictions startt+9m main thread CPU: 100% of one coret+9m replies/s falls below requests/s · backlog growingt+9m each eviction pass frees only a few KBt+10mmemory drops ~400 MB · evictions stopnote largest key is far smaller than the drop
Your call
What is driving the spikes?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.