Skip to content
All outages

Incident · Hard · Databases

Forty-three seconds

  • SEV1
  • 43-SECOND PARTITION
  • 24 H DEGRADED
90s

The situation

Replacing a piece of optical equipment cuts the link between your East and West coast data centres for 43 seconds. Your failover tool promotes West coast databases to primary. The link comes back. East still holds a few seconds of writes that West never received, and West has been taking new writes ever since.

The evidence

Replication lag during catch-upillustrative
  • lag
Logs and signalsReconstructed
22:52network: East ↔ West unreachable (43s)
22:53failover tool: quorum agrees → promote West primaries
22:54network restored
22:54East: writes present that were never replicated
22:55West: accepting new writes

Your call

What is the right call?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem