Step 1 · The pattern
Healthy, thanks to the cache
At peak, users send about 20 reads a second. The cache answers most of them, so the database sees only a trickle. Alone it could serve about 10 a second. The system is healthy, and depends on the cache staying warm.
- served
- cache miss → DB
- request
In short
Metastable failure: how it works and how it breaks
Some outages do not end when their cause does. A system can run well at a load it could not survive from a cold start: a warm cache hides most reads from a database that could never serve them all. That state is healthy but vulnerable. A short trigger, such as losing the cache, pushes the database past its limit. Answers arrive after the caller has timed out, so they are thrown away and never fill the cache, and clients retry, which adds load. The trigger is long gone and the loop keeps the system down. This is a metastable failure. Fixing the trigger does not help, because the trigger is no longer the problem. The way out is to break the loop: refuse most of the work, stop retries, let the useful work finish, then let traffic back in gradually.
How this comes up in interviews
"A failover blip overloaded a service. The blip was fixed in a minute, traffic is at its normal level, and an hour later requests still time out. Why?"
Then they ask
- What keeps the failure going once the trigger has gone, and how would you find that loop?
- Why does raising timeouts, or waiting for things to settle, not end it?
- How do you let traffic back in without tipping the system over again, and which metric do you watch?
A strong answer covers
- A system can run well at a load it could not survive from a cold start, such as a database hidden behind a warm cache.
- A short trigger pushes it past its limit. Late answers are thrown away and clients retry, which forms a loop that sustains itself.
- Fixing the trigger does not help, because the loop is now what holds the system down.
- Break the loop from outside: refuse most of the work, stop retries and let the useful work finish.
- Readmit traffic in steps while watching the metric that shows the loop, such as queue time or cache hit rate.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Bronson et al.: Metastable Failures in Distributed Systems (HotOS 2021) ↗ · Huang et al.: Metastable Failures in the Wild (OSDI 2022) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.