Incident · Easy · Redis / caching · Databases
Quiet Saturday, loud Monday
- SEV1
- MONDAY PEAK
- LOGINS FAILING
90s
The situation
On Saturday, to get a newly launched AI model in front of customers sooner, the refresh TTL on the cache of user settings is cut from 12 hours to 2. The weekend is quiet and nothing looks wrong. On Monday's peak, the core database cluster behind authentication and user management overloads. The TTL change is spotted quickly. Read load on the cluster keeps climbing anyway.
The evidence
- cache refresh writes
- read traffic
Logs and signalsReconstructed
Sat config: user-settings cache refresh TTL 12h → 2hSat db: load normal (weekend)Mon db core cluster: overloaded · auth requests failingMon suspect found: Saturday's TTL changeMon db: read volume still risingMon read traffic by client app version: newest builds far above older ones
Your call
The TTL change is real. What is the rest of the story?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.