Skip to content
Interview rounds

Observability · Production incident

One tenant, ten times the data

A noisy tenant sends 10x normal data. How do you contain it fairly?

Platform-SREStaff12 min
Interview-style scenarioLast reviewed: Checked against official documentation

Thursday 15:40 · metrics platformIllustrative scenario, not a real incident

Metric writes are failing for every tenant, and one tenant is sending ten times its normal volume

The metrics platform is shared by every team in the company. All of them write through one ingest gateway to a write path sized for normal load with some headroom.

Twenty minutes ago the search team's volume went up tenfold. Since then writes have failed for whoever happens to arrive, including teams that changed nothing.

What the dashboards and logs show

Search team's volume
10 times its usual rate
Every other tenant
normal volume, about a third of writes failing
Write path
saturated

Clarify · optional · scored under "Hypothesis before change"

Ask the interviewer

0 of 3 asked

Each question reveals a fact. What you ask before you act is part of your score.

First move · scored under "Mitigate before root cause"

Every team's writes are failing right now. What do you do first?

Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.