Skip to content
All outages

Incident · Medium · Redis / caching

More nodes, more outages

  • SEV1
  • 23:34 PDT
  • CAN'T OPEN FILES
90s

The situation

Users of a collaborative design tool cannot open files. One node of the managed, cluster-mode Redis that serves caching and message routing is pinned at 100% engine CPU. Redis command volume is steady and matches previous weeks. The team moves traffic to a new, larger cluster with more nodes. The problem comes back, and now it comes back more often.

The evidence

Engine CPU on the hot node vs command volumeillustrative
  • engine CPU
  • commands/s
Logs and signalsReconstructed
23:34redis node 3: engine CPU 100% · other nodes normal
23:40action: fail over node, launch larger cluster with more nodes
01:59new cluster: one node at 100% again
02:10slowlog: commands slow only after CPU is saturated
02:15no backup or snapshot running · no big keys found
note pub/sub traffic also runs on this clustered instance (moved there last month)
06:53one node at 100% again

Your call

What is pinning the node?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last reviewed: Official postmortem