Skip to content
All outages

Incident · Medium · Databases · Payments

Rows that exist, but don't

  • SEV1
  • 13:10
  • BANKING APP
90s

The situation

You run a bank. At 13:10 you add six nodes to your 21-node database cluster; a test with one node last week went fine. Soon customers see missing balances and transactions. Reads return 'not found' for records you know are there. At 13:24 the team finds and fixes a real bug in the card-network integration, and errors dip for a moment.

The evidence

'Row not found' errorsillustrative · nodes removed one by one from 14:00
  • not found
Logs and signalsReconstructed
13:10cluster: 6 new nodes joined ring (21 → 27)
13:10new nodes: auto_bootstrap=false (set Oct 2018)
13:12read key=acct_… → not found (quorum of empty replicas)
13:24card-network bug fixed · errors dip briefly
14:05node removed → errors step down

Your call

Why are existing rows 'not found'?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem