Skip to content

Deployments & config rollout · 1 of 8

Step 1 · The pattern

A fleet running one build

Users are served by a fleet of identical servers. One of them, the canary, takes only a small slice of traffic. A deploy system decides which build each server runs.

  • request
  • served

In short

Canary rollouts: how it works and how it breaks

Tests never cover everything, so some bad builds reach production. What you control is how many users meet one. If a release goes to every server at the same moment, a bad one fails every request, and it keeps failing until a person notices, decides and rolls back. A staged rollout puts a pause in the middle. The new build first goes to a canary: a small part of the fleet that takes a small slice of real traffic. An automated check compares the canary's errors with normal. If it looks healthy the release continues to wider stages; if not, it stops and the canary is put back, and most users never saw the new build. A canary only catches problems it can show in the time you give it, so it needs real traffic, the right signals and enough time.

How this comes up in interviews

"You deploy to a large fleet several times a day. How do you stop a bad build from reaching every user?"

Then they ask

  1. The canary passed and the build still caused an outage an hour later. What did the canary miss, and what do you change?
  2. What do you compare the canary against, and which signals decide pass or fail?
  3. Your first stage gets very little traffic. How do you know its result means anything?

A strong answer covers

  • Release in stages: a small canary on real traffic first, an automated check, then wider stages, with a way to stop at each one.
  • The goal is to limit how many users meet a bad build, since tests never cover everything.
  • A failed check halts the release and puts the old build back on the canary, without waiting for a person to notice.
  • Compare the canary with a control group on the old build started at the same time, on several signals and not error rate alone.
  • A canary only catches faults that show within its window, its traffic and its signals, so each stage needs enough time and requests.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Engineering write-up

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.