Databases & caching · Production incident
The primary's disk is filling, and no table grew
During a large backfill the PostgreSQL primary's disk heads towards full. How do you keep the primary up?
Tuesday 11:30 · orders database, PostgreSQLIllustrative scenario, not a real incident
db-primary: data volume 84% used, up from 42% at 09:00, and gaining about 1% every 4 minutes
The orders database is one PostgreSQL primary with two streaming replicas. Its 1,000 GB volume was at 42% this morning. A backfill that rewrites a 600 million row table started at 09:00 and is a little under half done.
Customer traffic is normal and nothing is failing. The alert fired at 80%, twenty minutes ago.
What the dashboards and logs show
- Primary volume
- 84% of 1,000 GB, 42% at 09:00
- Tables and indexes
- up 60 GB since 09:00
- Backfill
- running since 09:00, about 45% done
- Replicas
- two, both connected and streaming
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
The disk gains 1% every four minutes. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.