Incident · Hard · Redis / caching · Databases
The cache that never refills
- SEV1
- MORNING PEAK
- CLIENTS CAN'T BOOT
90s
The situation
During the morning peak, a routine rollout of a service-discovery agent restarts cache nodes a percentage at a time. At 25% the database tier melts and users cannot load the app. The restarted cache nodes are up and healthy. They are also empty, and they stay empty.
The evidence
- hit rate
- db load
Logs and signalsReconstructed
06:00agent rollout: 25% of cache nodes restarted → empty06:01cache miss: members of channel C… → query all shards06:01note: cache keyed by channel · table sharded by user06:02db: query timeout06:02cache fill: skipped (no result)
Your call
Why does the cache stay empty?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.