Skip to content
All outages

Incident · Medium · Databases · URL shortener · Redis

One channel, whole cluster

  • CHRONIC
  • 177 NODES
  • ON-CALL TOIL
90s

The situation

Your chat platform stores messages in a 177-node database cluster. Whenever a huge community posts an announcement, hundreds of thousands of people open the same channel at once. Latency rises for everyone, including small communities whose data lives elsewhere in the cluster. You keep adding nodes. It keeps happening.

The evidence

p99 read latency, whole clusterillustrative
  • p99
Logs and signalsReconstructed
20:00@everyone in a very large community
20:00reads: partition (channel, bucket) → same 3 replicas
20:01replicas: compaction falling behind · long GC pauses
20:01other queries on those nodes: slow
-- identical read issued 40,000× concurrently

Your call

What would actually fix it?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Engineering write-up