Observability · Design under changing requirements
A billion points a minute
Design a metrics pipeline that ingests billions of points a minute and still answers queries in seconds.
Design review · metrics platformIllustrative scenario, not a real incident
Dashboards time out at 09:00 every weekday, and ingestion falls behind while they load
Every service in the company sends its metrics to one time series database. Volume has grown tenfold in two years and is now about a billion points a minute.
Each morning, as people open their dashboards, queries over the last 30 days compete with ingestion on the same machine. Writes queue, alerts evaluate on stale data, and the dashboards time out anyway.
What the dashboards and logs show
- Ingest rate
- 1 billion points a minute, about 17 million a second
- Growth
- tenfold in two years, and still rising
- At 09:00
- query latency over 60 s, ingestion delay over 5 minutes
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
The redesign will take a quarter, and tomorrow at 09:00 the same thing will happen. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.