Skip to content
All outages

Incident · Hard · Redis / caching

Full every ten minutes

  • SLO BURN
  • EVERY FEW MINUTES
  • FOR MONTHS
90s

The situation

A large Redis instance used purely as an LRU cache has a strange heartbeat. Every several minutes, latency spikes and throughput drops for a few seconds, then everything is fine again. Client traffic is steady. On the memory graph, usage climbs to maxmemory, flat-tops, then suddenly falls by hundreds of megabytes. No key in the dataset is anywhere near that big.

The evidence

Memory used (% of maxmemory) vs slow requestsillustrative
  • memory
  • requests over 1s
Logs and signalsReconstructed
t+0m memory below maxmemory · evictions: 0
t+9m memory = maxmemory · evictions start
t+9m main thread CPU: 100% of one core
t+9m replies/s falls below requests/s · backlog growing
t+9m each eviction pass frees only a few KB
t+10mmemory drops ~400 MB · evictions stop
note largest key is far smaller than the drop

Your call

What is driving the spikes?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Engineering write-up