Skip to content

Consumer apps at scale · 10 of 13

Step 1 · The pattern

Every call starts with a name

App servers call the backend by name, so each call first asks the DNS resolver for its address, then calls. The resolver answers 3 lookups at a time, quickly. On a normal day, 6 calls a second barely register. (All numbers are illustrative.)

  • lookup, then call
  • call succeeded

In short

DNS resolver overload: how it works and how it breaks

Services inside a platform find each other by name, so almost every outgoing call starts with a DNS lookup. If nothing caches the answer near the caller, every call is a query to a small shared resolver that nobody thinks about. At a traffic peak the resolver is overwhelmed, lookups fail, the calls behind them fail, and retries add more lookups. The backend is healthy and unreachable. A trading app described record demand causing a "thundering herd" that brought down its DNS. Cache answers on each host for their TTL, so the resolver sees one lookup per name per TTL instead of one per call. Let one lookup refresh an expiring answer while others keep using it. And when the resolver cannot be reached, keep serving the last answer for a while, which RFC 8767 calls serving stale data.

What this simulation simplifies

  • One host cache stands for a cache on every host, and the backend is reached by a single name.
  • The resolver refuses beyond eight lookups. Real resolvers slow down, drop packets and time out.
  • In the fix a stale answer is served whenever the resolver is full. Real serve-stale applies only after the resolver fails to reach its upstream, within a limit.

How this comes up in interviews

"At peak, your internal services fail to reach each other with DNS errors, but every service is healthy. What is happening and how do you fix it?"

Then they ask

  1. Why didn't this show up in your load tests?
  2. What TTL would you choose for internal service names, and why?
  3. The resolver is down for a minute. What should callers do?

A strong answer covers

  • Every call starts with a lookup; with no cache near the caller, resolver load equals the call rate.
  • At peak the resolver is overwhelmed, calls fail before leaving the host, and retries add lookups.
  • Run a node-local DNS cache so lookups leave each host once per name per TTL.
  • Serve stale answers for a bounded time when the resolver fails (RFC 8767).
  • Measure resolver queries per second and include the resolvers in capacity planning.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.