Incident · Medium · Databases · URL shortener · Redis
One channel, whole cluster
- CHRONIC
- 177 NODES
- ON-CALL TOIL
90s
The situation
Your chat platform stores messages in a 177-node database cluster. Whenever a huge community posts an announcement, hundreds of thousands of people open the same channel at once. Latency rises for everyone, including small communities whose data lives elsewhere in the cluster. You keep adding nodes. It keeps happening.
The evidence
- p99
Logs and signalsReconstructed
20:00@everyone in a very large community20:00reads: partition (channel, bucket) → same 3 replicas20:01replicas: compaction falling behind · long GC pauses20:01other queries on those nodes: slow-- identical read issued 40,000× concurrently
Your call
What would actually fix it?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.