Step 1 · The pattern
Production, and the tools to fix it
Users reach the service through the edge and DNS, then the backbone that connects the data centres. Engineers manage the routers through a management network. Here that network, like remote access and internal DNS, rides on the same backbone: convenient, and one fewer thing to run. (Time is compressed.)
- user request
- served
In short
Out-of-band access: how it works and how it breaks
Engineers fix production through production: remote access, internal tools, internal DNS and dashboards all run over the same network as the service. Usually that is fine. When a change breaks that network itself, the people who could undo it are cut off too, and someone must travel to the site. One social network described a backbone change that disconnected its data centres; its primary and out-of-band network access were both down, so engineers went onsite, which took hours through the security designed to keep people out. Two habits prevent this. Apply risky network changes so they roll back automatically unless confirmed from a working session within minutes; if the change cut you off, it undoes itself. And keep a management network that is truly independent of production: separate links, its own DNS and its own credentials, tested regularly.
What this simulation simplifies
- Time is compressed: a site visit takes eight seconds here and hours in reality; the rollback timer is three seconds instead of minutes.
- The backbone is a single box. Real backbones are many routers, and a bad change can leave some paths working and others not.
- In the fix both protections are shown together. Either alone would have shortened the outage; the first to act restores the network.
How this comes up in interviews
"A change to your core network takes everything offline, including the tools your engineers use to fix it. How do you design so this cannot last for hours?"
Then they ask
- Your company already has an out-of-band network. Why might it fail at the same time?
- How does commit confirmed help when the change cuts off the person making it?
- What would you test in a game day for this scenario?
A strong answer covers
- Engineers usually fix production through production; a network change can remove every path at once.
- Apply risky changes so they roll back automatically unless confirmed from a working session.
- Keep a management network with its own links, DNS and credentials, sharing nothing with production.
- Validate changes before applying them and test the validators with known-bad changes.
- Rehearse the physical break-glass path and measure how long it really takes.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Meta engineering: More details about the October 4 outage (2021) ↗ · Junos OS documentation: Commit the configuration (commit confirmed) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.