Skip to content
All outages

Incident · Easy · Payments · Infra

Validation pending

  • SEV1
  • MARKET OPEN
  • VENDOR SYSTEM
90s

The situation

You run a retail stockbroker. Your own systems carry most of what clients do, but orders reach the exchanges through an order management system licensed from a vendor, and you cannot touch its internals. At the open, the busiest minutes of the day, that system develops connectivity issues. Orders pile up, each shown to its trader as “validation pending”. The vendor's team works on the pending queue, and it keeps growing.

The evidence

Orders: waiting in the queue vs pushed to the exchangesillustrative
  • waiting
  • pushed to exchanges
Logs and signalsReconstructed
market open: logins and order placement at their daily peak
vendor order system: connectivity issues
orders: validation pending, not yet at the exchange
vendor team: working on the pending queue
pending queue: still growing

Your call

Why does the queue keep growing while the vendor works on it?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem