Skip to content
All outages

Incident · Hard · Redis / caching · Databases

The cache that never refills

  • SEV1
  • MORNING PEAK
  • CLIENTS CAN'T BOOT
90s

The situation

During the morning peak, a routine rollout of a service-discovery agent restarts cache nodes a percentage at a time. At 25% the database tier melts and users cannot load the app. The restarted cache nodes are up and healthy. They are also empty, and they stay empty.

The evidence

Cache hit rate vs database loadillustrative
  • hit rate
  • db load
Logs and signalsReconstructed
06:00agent rollout: 25% of cache nodes restarted → empty
06:01cache miss: members of channel C… → query all shards
06:01note: cache keyed by channel · table sharded by user
06:02db: query timeout
06:02cache fill: skipped (no result)

Your call

Why does the cache stay empty?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem