Skip to content

DNS / HTTP · 8 of 8

Step 1 · The pattern

Three zones, sized for today

The service runs in three zones, which are separate data centres. Each zone has room for five requests at once and is handling about four. There is very little spare, which keeps the bill low.

  • request
  • answered

In short

Static stability: how it works and how it breaks

Most systems have two parts. The data plane does the everyday work, such as serving requests. The control plane makes changes, such as launching servers or updating configuration. The control plane has more moving parts and is more likely to be struggling during a large failure. A lean service sized for exactly today's traffic plans to react when a zone fails: detect it, then launch replacements elsewhere. That plan makes recovery depend on the control plane at the worst possible moment, and users see errors for as long as it takes. A statically stable service is ready before anything breaks. With three zones it runs half as much capacity again, so each zone sits about two-thirds full and any two can carry everything. When a zone fails, health checks move traffic and nothing else has to happen. The price is paying for idle capacity all year.

How this comes up in interviews

"Your service runs in three zones and one zone fails. Describe what has to happen for users to notice nothing."

Then they ask

  1. Why is launching replacement servers during the failure a weak recovery plan?
  2. How much capacity do you run in each zone, and what does that cost you all year?
  3. What is the difference between the control plane and the data plane, and which is likelier to be impaired in a large failure?

A strong answer covers

  • The data plane does everyday work such as serving requests. The control plane makes changes such as launching servers.
  • Reacting to a failure by launching capacity makes recovery depend on the control plane at the worst moment.
  • A statically stable service is ready before anything breaks: recovery needs no launch and no change.
  • With three zones, run half as much capacity again so each sits about two-thirds full and any two carry everything.
  • Health checks move traffic away from the failed zone. The price is paying for idle capacity all year.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Engineering write-up

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.