Step 1 · The pattern
A build and a switch
A deploy system decides which build each server runs: it can push a new one or roll back. A flag is a separate switch that every server reads. Rolling back changes the build. It never moves the flag.
- request
- served
- build change / flag on
In short
A rollback that makes it worse: how it works and how it breaks
When something breaks after a deploy, the reflex is to roll back, and most of the time that works. But a rollback only restores code. It leaves everything else that changed where it is: a flag that is still on, data already written in a new shape, a saved snapshot that already holds the fault. So first check where the fault lives. Here a deploy missed one server, and a flag was then switched on. The new build handles the flag correctly. On the old build the same flag wakes up retired code, so the missed server misbehaves. Rolling back puts the healthy servers on the old build with the flag still on, and every server now runs the retired code. Comparing the bad server with the good ones shows the release was the cure. Turn the flag off, then finish the deploy.
How this comes up in interviews
"After a release, a flag is switched on and one server starts misbehaving while the rest are fine. The team wants to roll back. What do you check first?"
Then they ask
- What does a rollback leave untouched, and how can that make the old build unsafe to return to?
- How would you test, before shipping, that a release can be rolled back at all?
- How do you detect that a deploy missed a host before a flag change exposes it?
A strong answer covers
- A rollback restores code only. A flag that is on, data written in a new shape and saved snapshots stay where they are.
- Compare the bad server with the good ones first: if it is the one the deploy missed, the release is the cure and not the cause.
- Rolling back then moves every healthy server onto the old build with the flag still on, which spreads the fault.
- Turn the flag off to stop the damage, then roll forward by finishing the deploy.
- Verify each deploy by asking every host what it runs, and never give an old flag name a new meaning.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: SEC order: Knight Capital Americas LLC (2013, PDF) ↗ · Azure Front Door post-incident review (October 2025) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.