Step 1 · The pattern
A file nobody types
A generator rebuilds a config file every few seconds from live data. A publisher sends each new version to every proxy. The proxies load it and use it to serve users. No person reads the file on the way.
- request
- served
- config file
In short
Validate config, keep last known good: how it works and how it breaks
Many systems run on files that a program writes: rule sets, feature lists, allow lists. A generator rebuilds the file on a schedule and ships it to every server, with no person in the loop. The generator's inputs can change, so one day the file is twice the size, half empty or malformed. If servers trust whatever arrives and crash on a file they cannot load, one bad file takes down the whole fleet in seconds. Two defences work together. Check the file before publishing it, using the same limits and the same loading code the servers use, and refuse to ship a file that fails. Then make each server careful too: if a new file will not load, reject it, raise an alert and keep serving with the last file that worked. The config goes a little stale. Nobody goes down.
How this comes up in interviews
"A generator rebuilds a config file and ships it to every server with no person in the loop. One day the file is bad. How do you make sure nobody goes down?"
Then they ask
- A file passes your publish check and still crashes the servers. How can that happen, and how do you close the gap?
- Servers keep the last good file. How stale may it get, and how would you know the fleet is refusing new files?
- A server restarts while the newest file is bad, so it has no last good file in memory. What should it do?
A strong answer covers
- A generated file has no person in the loop and its inputs can change, so treat every new version as a deploy.
- Check the file before publishing with the same limits and loading code the servers use, and refuse to ship one that fails.
- No check is complete, so each server rejects a file it cannot load, raises an alert and keeps serving with the last file that worked.
- A check only protects against faults it shares with the loader: testing syntax and size is not the same as loading the file.
- The price is stale config, which needs an alert on its age and a decision per file on how old is too old.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Cloudflare outage on November 18, 2025 ↗ · CrowdStrike Channel File 291 root cause analysis (PDF, Aug 2024) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.