Skip to content
All outages

Incident · Hard · Payments · Redis

Charged again. And again.

  • SEV1
  • BILLING
  • SUPPORT ON FIRE
90s

The situation

Customers report their credit cards being charged over and over for account top-ups. The payments team insists the charges are "correct" given the data they see.

The evidence

Avg account balance vs rechargesillustrative
  • balance
  • auto-recharges
Logs and signalsReconstructed
t-10mbrief network blip
t-9m all redis replicas: requesting full resync
t-8m redis master: overloaded
t-6m ops: restarting redis master
t-5m redis: loading config … AOF file not found
t-5m balance:acct_* = 0
t-4m billing: balance < threshold → recharge card

Your call

Where's the real bug?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem