Step 1 · The pattern
One fleet, two ways in
Edge machines serve every user. Changes reach them by two roads: code releases, which go out in stages, and config (rules, flags, data files), which lands on every machine within seconds.
- request
- served
In short
Staged config rollouts: how it works and how it breaks
Most teams release code carefully: one machine, a check, then more. Configuration often takes another road. Rules, flags and data files are pushed through a fast channel that reaches every machine in seconds, because most of these changes are small and many are urgent. But a machine does not care which road a change arrived by. A rule that burns CPU on every request, or a file the software cannot parse, breaks a machine as surely as a bad binary, and the fast channel delivers it to the whole fleet before any signal can come back. The fix is to give config the same treatment as code: validate it, send it to a small part of the fleet first, watch CPU and errors, and continue only when they stay normal. Keep the instant path for real emergencies, as a deliberate exception.
How this comes up in interviews
"Your code ships in stages, but a rule edited in a dashboard is live on every machine within seconds. What can go wrong, and how would you redesign it?"
Then they ask
- Some changes are urgent, such as blocking an attack. How do you keep a fast path without it becoming the default?
- The bad rule has broken the machines that also serve your config console. How do you revert it?
- Which signals should halt a config push automatically, and what does a revert consist of?
A strong answer covers
- Risk depends on how fast a change reaches every machine and whether anything can stop it, not on whether it is called code or config.
- Software interprets config, so a rule that burns CPU or a file that will not parse breaks a machine as surely as a bad binary.
- Validate the change, send it to a small part of the fleet first, watch CPU and errors, and continue only when they stay normal.
- Keep the instant path as a deliberate exception for real emergencies, still with a small first ring and an automatic halt.
- Version every change so a revert re-activates the previous version, over a path the bad config cannot break.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Details of the Cloudflare outage on July 2, 2019 ↗ · Holistic Configuration Management at Facebook (SOSP 2015) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.