Incident · Easy · Payments
Seven of eight
- SEV0
- MARKET OPEN
- 45 MINUTES
90s
The situation
The market opens and your order router starts flooding the exchange with trades nobody asked for. You are losing millions a minute. New code went out to the eight router servers this week. Before the open, 97 automated emails mentioning an old, retired feature being 'disabled' went unread. The room agrees: roll back the release.
The evidence
- loss
Logs and signalsReconstructed
pre-open97 emails: 'Power Peg disabled' (unread)09:30servers 1–7: new code · flag = new order type09:30server 8: old code · same flag = retired test feature09:31server 8: sending orders in a loop09:40ops: uninstall new code from servers 1–7?
Your call
What happens if you roll back the seven servers?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.