Practice
SEV-1 simulationCaching and state · Real outage · Reconstructed data · 15–20 minutes
Incident #019 — Charged again. And again.
Customer impact
Customers with auto-recharge are being charged repeatedly. Balances read as zero and cannot be updated.
| Signal | Current value |
|---|---|
| Card recharges | repeating per account |
| Balances in Redis | reading as 0 |
| redis-master | read-only |
| Accounts affected | ≈1.1% |
| Voice and SMS | working |
| Bookkeeping database | balances intact |
Your task
Monitoring has just flagged an anomaly in billing. Work out what is happening, using the service map, metrics, logs, traces and recent changes. Stop the harm first, then name the cause and the permanent fix.
03:30Impact continuing · +0 minabout 0 wrong card charges lost since you joined
Investigate · each first look costs 2 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time