Skip to content
All outages

Incident · Medium · Redis / caching · URL shortener

The fix that kept it down

  • SEV1
  • ~2.5 HOURS
  • GLOBAL
90s

The situation

Someone changes one configuration value. An automated checker decides the cached copy is invalid and sends every client to the database for a fresh one. The database cluster buckles under hundreds of thousands of queries a second. The team spots the bad value and corrects it. The site stays down.

The evidence

Database queries per secondillustrative
  • queries/s
Logs and signalsReconstructed
t+0 config value changed → clients read it as invalid
t+0 auto-fix: fetch a fresh value from the database
t+1m db: overloaded, returning errors
t+1m client: db error → delete cache key
t+9m config value corrected
t+10mdb: still overloaded

Your call

Why is it still down after the value was fixed?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem