Skip to content
All outages

Incident · Easy · Redis / caching · Databases

Quiet Saturday, loud Monday

  • SEV1
  • MONDAY PEAK
  • LOGINS FAILING
90s

The situation

On Saturday, to get a newly launched AI model in front of customers sooner, the refresh TTL on the cache of user settings is cut from 12 hours to 2. The weekend is quiet and nothing looks wrong. On Monday's peak, the core database cluster behind authentication and user management overloads. The TTL change is spotted quickly. Read load on the cluster keeps climbing anyway.

The evidence

Core database load: cache rewrites vs readsillustrative
  • cache refresh writes
  • read traffic
Logs and signalsReconstructed
Sat config: user-settings cache refresh TTL 12h → 2h
Sat db: load normal (weekend)
Mon db core cluster: overloaded · auth requests failing
Mon suspect found: Saturday's TTL change
Mon db: read volume still rising
Mon read traffic by client app version: newest builds far above older ones

Your call

The TTL change is real. What is the rest of the story?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem