Skip to content

Resilience patterns · 4 of 9

Step 1 · The pattern

Three layers, each with retries

A tap in the app goes to a gateway, then a service, then the database. Calls fail now and then, so each layer retries a failed call. Here a rare database error is retried by the service and the user never sees it.

  • request
  • call failed
  • retry
  • answered

In short

Retry storms across layers: how it works and how it breaks

A user request usually crosses several layers: an app calls a gateway, the gateway calls a service, the service calls a database. Each team adds retries to its own layer, and each choice looks sensible alone. Together they multiply. If the database fails a call, the service tries three times. The gateway sees one failure and runs the service three times. The app does the same to the gateway. One tap becomes up to 27 database calls, sent at the moment the database can least afford them. The extra load keeps it failing after the first cause has gone. The fix is a rule for the whole call chain: retry at one layer only, usually the one next to the failure, and have every other layer pass the error up. Give that layer a retry budget, so retries stay a small share of normal traffic however bad things get.

How this comes up in interviews

"A request crosses an app, a gateway and a service before the database, and each layer retries a failed call. What happens when the database slows down?"

Then they ask

  1. Would lowering the number of tries at every layer solve it? If not, what does?
  2. Which layer should own the retry, and how do the others know not to retry?
  3. How does a retry budget work, and why is backoff with jitter not a substitute for it?

A strong answer covers

  • Tries multiply down the chain: three layers of three tries can turn one request into 27 database calls.
  • The extra load lands when the database can least afford it and keeps it overloaded after the first cause has gone.
  • Retry at one layer only, usually the one next to the failure, and have every other layer pass the error up.
  • Give that layer a retry budget, so retries stay a small share of normal traffic however bad things get.
  • Backoff and jitter spread the calls out in time but do not reduce how many are sent.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.