Observability · Design under changing requirements
The slow trace that was never kept
When would you use tail-based sampling over head-based, and what does each lose?
Wednesday 10:15 · incident reviewIllustrative scenario, not a real incident
Checkout p99 crossed 8 seconds for ten minutes, and no trace of a slow request can be found
Every service traces with the same SDK settings, and one trace in ten ends up in the trace store.
During last night's latency spike about one request in two hundred was slow. The team has plenty of traces from that window, and nearly all of them are fast, healthy requests.
What the dashboards and logs show
- Sampling
- 10% of traces are stored
- Traces kept from the spike
- thousands, nearly all under 200 ms
- Customer's trace ID
- not found in the trace store
- Tracing budget
- fixed: storage cannot grow much
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
The spike may return tonight, and the new design will take weeks. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.