Incident · Hard · Databases
Forty-three seconds
- SEV1
- 43-SECOND PARTITION
- 24 H DEGRADED
90s
The situation
Replacing a piece of optical equipment cuts the link between your East and West coast data centres for 43 seconds. Your failover tool promotes West coast databases to primary. The link comes back. East still holds a few seconds of writes that West never received, and West has been taking new writes ever since.
The evidence
- lag
Logs and signalsReconstructed
22:52network: East ↔ West unreachable (43s)22:53failover tool: quorum agrees → promote West primaries22:54network restored22:54East: writes present that were never replicated22:55West: accepting new writes
Your call
What is the right call?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.