Step 1 · The pattern
One place to find everything
Every user request asks the coordinator where its service is, then calls it. The monitoring asks the coordinator which services to check, then checks them. One shared system makes discovery and configuration easy for every team. (All numbers are illustrative.)
- user request
- monitoring check
- answered
In short
The dependency everything shares: how it works and how it breaks
A coordination service for service discovery, configuration or leader election is convenient, so over time everything uses it. Requests find their backends through it, deploy tools read it, and the monitoring discovers what to watch through it. When it degrades, every service fails at the same moment, and the dashboards that should explain the outage fail with them. An online game platform described exactly this: a three-day outage in which critical monitoring relied on the affected coordination service. Make the request path statically stable: services keep the last known addresses and configuration and keep serving when the coordinator cannot answer, so it is needed for changes, not for every request. Give the monitoring its own path that does not depend on what it watches. And split one shared cluster into several, so a failure takes out one domain, not all of them.
What this simulation simplifies
- One coordinator box stands for a whole cluster, and its failure is all-or-nothing timeouts. Real degradation is often slow responses that get worse under load.
- The gateway caches addresses for every service. Real services cache discovery and configuration in different ways, and some read the coordinator on every call.
- Monitoring is one box with one target list. Real observability stacks have several parts, each with its own dependencies.
How this comes up in interviews
"Your service discovery and configuration live in one cluster that everything uses. How do you keep an outage of that cluster from taking down the whole product?"
Then they ask
- During the outage your dashboards were empty. Why, and what do you change?
- What should a service do when it cannot reach discovery at start-up?
- Would you run one cluster or several, and how would you split them?
A strong answer covers
- Shared coordination ends up on every request path, so its failure takes everything down at once.
- Make the data path statically stable: keep and serve the last known addresses and configuration.
- Monitoring, alerting and paging must not depend on the systems they watch.
- Split shared clusters by domain to limit the blast radius, and limit new uses.
- Practise coordinator loss and restore in a game day before it happens for real.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Roblox: Roblox return to service 10/28-10/31 2021 ↗ · Amazon Builders' Library: Static stability using Availability Zones ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.