Incident · Easy · Redis / caching
Failed over. Nobody followed.
- SEV2
- 14:01 PDT
- PARTIAL API OUTAGE
90s
The situation
The cloud provider automatically migrates the VM that hosts the primary of your highly available Redis cluster, and the node drops off the network. Within a minute the cluster has promoted the secondary. Ten minutes later some API instances are still failing their Redis calls, while others are fine.
The evidence
- cluster has a primary
- API errors
Logs and signalsReconstructed
14:01alert: API anomalies · redis primary unreachable14:02redis: secondary promoted to primary14:02cause: automatic VM migration by cloud provider14:03api-12: redis calls OK14:03api-07: redis calls failing (timeouts)14:11decision: rolling restart of all API instances
Your call
The failover worked. Why is part of the API still failing?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.