Observability · Production incident
The label that tripled the bill
Cardinality exploded after a deploy and costs tripled overnight. How do you find the label and stop it?
Thursday 09:00 · metrics platformIllustrative scenario, not a real incident
Active series are 3.2 times last week's, and the metrics bill forecast has tripled
The platform normally holds 40,000 active series. Since yesterday evening the count has climbed in a straight line and has not levelled off.
Nothing is failing. Dashboards load, alerts evaluate, traffic is ordinary for a Thursday. The only deploy in the window was checkout 4.12 at 18:30, described as a logging and metrics clean-up.
What the dashboards and logs show
- Active series
- 130,000 and rising by about 6,000 an hour
- Memory limit
- reached at about 150,000 series
- Traffic
- flat against last Thursday
- Retention
- 15 days (the default)
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
At this rate the memory limit is about three hours away. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.