Skip to content
All outages

Incident · Medium · Consensus · Kafka

Five nodes, one building

  • SEV1
  • 19:48 ET
  • TRADING HALTED
90s

The situation

A cooling failure shuts down racks in one data hall of your cloud provider. Your exchange runs across several zones, and losing one zone is a failure it was designed to survive. Yet nearly all trading halts. The matching engine is a five-node replicated cluster, kept close together inside one placement group so that votes travel fast. The provider then terminates the instances in that hall.

The evidence

Matching-engine nodes able to voteillustrative
  • voting nodes
  • needed for a majority
Logs and signalsReconstructed
19:20provider: cooling lost in one data hall, racks shut down
19:48nearly all trading halted
21:29provider terminates instances in the placement group
21:29matching engine: 2 of 5 nodes reachable
21:30other zones: healthy, instances running
21:31event stream: two broker clusters stuck healing, producers cannot write

Your call

Why did losing one data hall stop an exchange that spans several zones?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last reviewed: Official postmortem