Incident · Medium · Redis / caching · URL shortener
The fix that kept it down
- SEV1
- ~2.5 HOURS
- GLOBAL
90s
The situation
Someone changes one configuration value. An automated checker decides the cached copy is invalid and sends every client to the database for a fresh one. The database cluster buckles under hundreds of thousands of queries a second. The team spots the bad value and corrects it. The site stays down.
The evidence
- queries/s
Logs and signalsReconstructed
t+0 config value changed → clients read it as invalidt+0 auto-fix: fetch a fresh value from the databaset+1m db: overloaded, returning errorst+1m client: db error → delete cache keyt+9m config value correctedt+10mdb: still overloaded
Your call
Why is it still down after the value was fixed?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.