Skip to content

Deployments & config rollout · 6 of 8

Step 1 · The pattern

One side writes, one side reads

The control plane (two instances, CP A and CP B) turns each edit into config and sends it to every node. The data plane is those nodes: they load the config and serve users with it. Every box runs a build, and builds change one box at a time.

  • request / edit
  • config
  • served

In short

Version skew: control vs data plane: how it works and how it breaks

Large systems split into a control plane, which accepts changes and turns them into config, and a data plane, the machines that serve traffic using that config. The two are upgraded separately and never in the same instant, so during every upgrade several builds of each are running. That gap is version skew. Whatever one side writes must be readable by every build of the other side that is still alive, and two control-plane builds must agree with each other about what they write. Each change can be valid by itself and the combination can still break the readers. Because config reaches every node, the damage is as wide as the old build. The safe order is readers first, writers last: teach every node to read the new format, confirm they all report it, and only then let the control plane start writing it.

How this comes up in interviews

"You are adding a new config format that the control plane writes and every data-plane node must read. How do you roll it out safely?"

Then they ask

  1. Both formats pass validation, yet nodes crash on the new one. Why did validation not catch it?
  2. After the writer switches to the new format, the nodes are rolled back. What happens, and how do you plan for it?
  3. How does the control plane know that every live node can read the new format?

A strong answer covers

  • Control plane and data plane are upgraded separately, so several builds of each run together during every upgrade.
  • Whatever one side writes must be readable by every build of the other side still alive. A validator checks form, not who can read it.
  • Order the rollout readers first, writers last: teach every node to read the new format before anything writes it.
  • The new writer keeps producing the old format until every node reports a build that reads the new one.
  • Config reaches every node, so the damage is as wide as the old build, and rolling readers back recreates the skew.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.