Practice
SEV-1 simulationDatabases · Real outage · Reconstructed data · 15–20 minutes
Incident #013 — The database stopped writing
Customer impact
No new events can be stored. The database is refusing writes, and the event backlog is growing out of control.
| Signal | Current value |
|---|---|
| Database writes | refused |
| XIDs left before the limit | under 1 million |
| Wraparound vacuums | 1 table left |
| Event backlog | growing fast |
| Reads | served by replicas |
Your task
You join after nearly three hours of waiting on vacuums. Work out why the database stopped writing and what is still blocking it, using the service map, metrics, logs, traces and configuration. Choose how to restore writes, then name the cause and the permanent fix.
14:00Impact continuing · +0 minabout 0 thousand events not stored lost since you joined
Investigate · each first look costs 2 simulated minutes
What do you want to look at?
Nothing open yet. Pick a view above. You do not have to open them all before you act.
Step 1 · mitigate · you can act at any time