Skip to content
All concepts

Concept

Metastable failure

A metastable failure is an outage that keeps itself going after the thing that triggered it has been fixed. Something in the overloaded state, usually retries or cold caches, creates enough extra work to hold the system there. Recovery needs the loop broken from outside: shed load, stop retries, then let traffic back gradually.

Watch the trigger end while the system stays down, held there by its own retries. A live step from the lesson Metastable failure (Resilience patterns). Timings and counts are illustrative.

Where you'll see it

Sources: Bronson et al.: Metastable Failures in Distributed Systems (HotOS 2021) ↗ · Huang et al.: Metastable Failures in the Wild (OSDI 2022) ↗ · Report a problem

The definition is in our own words; the sources are those of the lesson Metastable failure.