Incident · Hard · Payments · Redis
Charged again. And again.
- SEV1
- BILLING
- SUPPORT ON FIRE
90s
The situation
Customers report their credit cards being charged over and over for account top-ups. The payments team insists the charges are "correct" given the data they see.
The evidence
- balance
- auto-recharges
Logs and signalsReconstructed
t-10mbrief network blipt-9m all redis replicas: requesting full resynct-8m redis master: overloadedt-6m ops: restarting redis mastert-5m redis: loading config … AOF file not foundt-5m balance:acct_* = 0t-4m billing: balance < threshold → recharge card
Your call
Where's the real bug?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.