Observability · Production incident
Twenty minutes behind
A Kafka-backed ingest pipeline lags by 20 minutes. What do you check, in order?
Wednesday 14:05 · metrics pipelineIllustrative scenario, not a real incident
Metrics are 20 minutes stale: consumer lag on metrics-ingest is rising
Collectors publish samples to the metrics-ingest topic. A group of writers consumes them and writes each batch to the time series database.
For the last half hour the writers have fallen further and further behind. Dashboards show data from 20 minutes ago, and alerts are evaluating the past. Nothing has crashed.
What the dashboards and logs show
- Consumer lag
- 20 minutes behind, and rising
- Topic
- metrics-ingest, 6 partitions
- Writers
- 6 running
- Brokers
- no under-replicated partitions
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Alert rules are evaluating data that is 20 minutes old. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.