Incident · Medium · Consensus · Kafka
Five nodes, one building
- SEV1
- 19:48 ET
- TRADING HALTED
90s
The situation
A cooling failure shuts down racks in one data hall of your cloud provider. Your exchange runs across several zones, and losing one zone is a failure it was designed to survive. Yet nearly all trading halts. The matching engine is a five-node replicated cluster, kept close together inside one placement group so that votes travel fast. The provider then terminates the instances in that hall.
The evidence
- voting nodes
- needed for a majority
Logs and signalsReconstructed
19:20provider: cooling lost in one data hall, racks shut down19:48nearly all trading halted21:29provider terminates instances in the placement group21:29matching engine: 2 of 5 nodes reachable21:30other zones: healthy, instances running21:31event stream: two broker clusters stuck healing, producers cannot write
Your call
Why did losing one data hall stop an exchange that spans several zones?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.