Databases & caching · On-call judgement
03:10, and a replica is 90 seconds behind
An alert wakes you at night because a read replica is 90 seconds behind the primary. Is it worth a page, and what should the alert measure?
Wednesday 03:10 · platform on-call, accounts databaseIllustrative scenario, not a real incident
ReplicaLagHigh: pg-replica-2 is 90 seconds behind, threshold 30 seconds
The pager has woken you. The accounts database has a primary and two read replicas, and the alert fires when either replica is more than 30 seconds behind.
The nightly batch that rebuilds account summaries started at 02:00. This alert has paged three times this month between 02:00 and 04:00, and each time it cleared without anyone doing anything.
What the dashboards and logs show
- Alert
- replica lag 90 s on pg-replica-2, firing since 03:04
- Nightly batch
- started 02:00, usually done by 05:00
- Error rate, API
- normal
- This alert, this month
- 3 pages, 0 actions taken
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
You are awake with the alert open. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.