Incident · Medium · Redis / caching
More nodes, more outages
- SEV1
- 23:34 PDT
- CAN'T OPEN FILES
90s
The situation
Users of a collaborative design tool cannot open files. One node of the managed, cluster-mode Redis that serves caching and message routing is pinned at 100% engine CPU. Redis command volume is steady and matches previous weeks. The team moves traffic to a new, larger cluster with more nodes. The problem comes back, and now it comes back more often.
The evidence
- engine CPU
- commands/s
Logs and signalsReconstructed
23:34redis node 3: engine CPU 100% · other nodes normal23:40action: fail over node, launch larger cluster with more nodes01:59new cluster: one node at 100% again02:10slowlog: commands slow only after CPU is saturated02:15no backup or snapshot running · no big keys foundnote pub/sub traffic also runs on this clustered instance (moved there last month)06:53one node at 100% again
Your call
What is pinning the node?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.