Skip to content
All outages

Incident · Easy · Payments

Seven of eight

  • SEV0
  • MARKET OPEN
  • 45 MINUTES
90s

The situation

The market opens and your order router starts flooding the exchange with trades nobody asked for. You are losing millions a minute. New code went out to the eight router servers this week. Before the open, 97 automated emails mentioning an old, retired feature being 'disabled' went unread. The room agrees: roll back the release.

The evidence

Cumulative lossillustrative
  • loss
Logs and signalsReconstructed
pre-open97 emails: 'Power Peg disabled' (unread)
09:30servers 1–7: new code · flag = new order type
09:30server 8: old code · same flag = retired test feature
09:31server 8: sending orders in a loop
09:40ops: uninstall new code from servers 1–7?

Your call

What happens if you roll back the seven servers?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last reviewed: Regulator