Observability · Production incident
One tenant, ten times the data
A noisy tenant sends 10x normal data. How do you contain it fairly?
Thursday 15:40 · metrics platformIllustrative scenario, not a real incident
Metric writes are failing for every tenant, and one tenant is sending ten times its normal volume
The metrics platform is shared by every team in the company. All of them write through one ingest gateway to a write path sized for normal load with some headroom.
Twenty minutes ago the search team's volume went up tenfold. Since then writes have failed for whoever happens to arrive, including teams that changed nothing.
What the dashboards and logs show
- Search team's volume
- 10 times its usual rate
- Every other tenant
- normal volume, about a third of writes failing
- Write path
- saturated
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Every team's writes are failing right now. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.