Skip to content
Practice

SEV-1 simulationCaching and state · Real outage · Reconstructed data · 15–20 minutes

Incident #019 — Charged again. And again.

Customer impact

Customers with auto-recharge are being charged repeatedly. Balances read as zero and cannot be updated.

SignalCurrent value
Card rechargesrepeating per account
Balances in Redisreading as 0
redis-masterread-only
Accounts affected≈1.1%
Voice and SMSworking
Bookkeeping databasebalances intact

Your task

Monitoring has just flagged an anomaly in billing. Work out what is happening, using the service map, metrics, logs, traces and recent changes. Stop the harm first, then name the cause and the permanent fix.

03:30Impact continuing · +0 minabout 0 wrong card charges lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

Cards are being charged again and again. What do you do first?