Skip to content

Deployments & config rollout · 2 of 8

Step 1 · The pattern

One fleet, two ways in

Edge machines serve every user. Changes reach them by two roads: code releases, which go out in stages, and config (rules, flags, data files), which lands on every machine within seconds.

  • request
  • served

In short

Staged config rollouts: how it works and how it breaks

Most teams release code carefully: one machine, a check, then more. Configuration often takes another road. Rules, flags and data files are pushed through a fast channel that reaches every machine in seconds, because most of these changes are small and many are urgent. But a machine does not care which road a change arrived by. A rule that burns CPU on every request, or a file the software cannot parse, breaks a machine as surely as a bad binary, and the fast channel delivers it to the whole fleet before any signal can come back. The fix is to give config the same treatment as code: validate it, send it to a small part of the fleet first, watch CPU and errors, and continue only when they stay normal. Keep the instant path for real emergencies, as a deliberate exception.

How this comes up in interviews

"Your code ships in stages, but a rule edited in a dashboard is live on every machine within seconds. What can go wrong, and how would you redesign it?"

Then they ask

  1. Some changes are urgent, such as blocking an attack. How do you keep a fast path without it becoming the default?
  2. The bad rule has broken the machines that also serve your config console. How do you revert it?
  3. Which signals should halt a config push automatically, and what does a revert consist of?

A strong answer covers

  • Risk depends on how fast a change reaches every machine and whether anything can stop it, not on whether it is called code or config.
  • Software interprets config, so a rule that burns CPU or a file that will not parse breaks a machine as surely as a bad binary.
  • Validate the change, send it to a small part of the fleet first, watch CPU and errors, and continue only when they stay normal.
  • Keep the instant path as a deliberate exception for real emergencies, still with a small first ring and an automatic halt.
  • Version every change so a revert re-activates the previous version, over a path the bad config cannot break.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Official postmortem

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.