Skip to content
Interview rounds

Observability · Production incident

The label that tripled the bill

Cardinality exploded after a deploy and costs tripled overnight. How do you find the label and stop it?

Platform-SRESenior12 min
Interview-style scenarioLast reviewed: Checked against official documentation

Thursday 09:00 · metrics platformIllustrative scenario, not a real incident

Active series are 3.2 times last week's, and the metrics bill forecast has tripled

The platform normally holds 40,000 active series. Since yesterday evening the count has climbed in a straight line and has not levelled off.

Nothing is failing. Dashboards load, alerts evaluate, traffic is ordinary for a Thursday. The only deploy in the window was checkout 4.12 at 18:30, described as a logging and metrics clean-up.

What the dashboards and logs show

Active series
130,000 and rising by about 6,000 an hour
Memory limit
reached at about 150,000 series
Traffic
flat against last Thursday
Retention
15 days (the default)

Clarify · optional · scored under "Hypothesis before change"

Ask the interviewer

0 of 3 asked

Each question reveals a fact. What you ask before you act is part of your score.

First move · scored under "Mitigate before root cause"

At this rate the memory limit is about three hours away. What do you do first?

Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.