Incident · Medium · Databases · Payments
Rows that exist, but don't
- SEV1
- 13:10
- BANKING APP
90s
The situation
You run a bank. At 13:10 you add six nodes to your 21-node database cluster; a test with one node last week went fine. Soon customers see missing balances and transactions. Reads return 'not found' for records you know are there. At 13:24 the team finds and fixes a real bug in the card-network integration, and errors dip for a moment.
The evidence
- not found
Logs and signalsReconstructed
13:10cluster: 6 new nodes joined ring (21 → 27)13:10new nodes: auto_bootstrap=false (set Oct 2018)13:12read key=acct_… → not found (quorum of empty replicas)13:24card-network bug fixed · errors dip briefly14:05node removed → errors step down
Your call
Why are existing rows 'not found'?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.