Skip to content
Practice

SEV-1 simulationDatabases · Real outage · Reconstructed data · 15–20 minutes

Incident #013 — The database stopped writing

Customer impact

No new events can be stored. The database is refusing writes, and the event backlog is growing out of control.

SignalCurrent value
Database writesrefused
XIDs left before the limitunder 1 million
Wraparound vacuums1 table left
Event backloggrowing fast
Readsserved by replicas

Your task

You join after nearly three hours of waiting on vacuums. Work out why the database stopped writing and what is still blocking it, using the service map, metrics, logs, traces and configuration. Choose how to restore writes, then name the cause and the permanent fix.

14:00Impact continuing · +0 minabout 0 thousand events not stored lost since you joined

Investigate · each first look costs 2 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

All but one table has been vacuumed, and writes are still refused. What do you do?