Skip to content
All outages

Incident · Easy · Redis / caching

Failed over. Nobody followed.

  • SEV2
  • 14:01 PDT
  • PARTIAL API OUTAGE
90s

The situation

The cloud provider automatically migrates the VM that hosts the primary of your highly available Redis cluster, and the node drops off the network. Within a minute the cluster has promoted the secondary. Ten minutes later some API instances are still failing their Redis calls, while others are fine.

The evidence

API error rate vs Redis cluster healthillustrative
  • cluster has a primary
  • API errors
Logs and signalsReconstructed
14:01alert: API anomalies · redis primary unreachable
14:02redis: secondary promoted to primary
14:02cause: automatic VM migration by cloud provider
14:03api-12: redis calls OK
14:03api-07: redis calls failing (timeouts)
14:11decision: rolling restart of all API instances

Your call

The failover worked. Why is part of the API still failing?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem