Skip to content

Observability · 1 of 6

Step 1 · The pattern

One page, thirty calls

Each page a user opens makes 30 calls, spread over three servers. Every call comes back in about 20 ms, and the dashboard shows that one typical number.

  • call
  • fast answer

In short

p50 vs p99: how it works and how it breaks

The median, or p50, is the latency of the typical request: half are faster. The 99th percentile, p99, is the latency that only one request in a hundred exceeds. They answer different questions. If one server in three is slow for a small share of its calls, one call in fifty can take ten times longer while the median stays where it was and the average rises by a few milliseconds. A dashboard showing only those two stays green. Users do not experience single calls, though. A page that makes thirty calls needs all thirty to be fast, so a slow two percent of calls reaches almost half of the page loads. Read a high percentile next to the median, look at the shape of the distribution and not one number, and split it by server to see where the tail comes from.

How this comes up in interviews

"Median latency is flat and the average has barely moved, yet users say pages often feel slow. How do you investigate?"

Then they ask

  1. Each page makes thirty calls. How does a slow two percent of calls turn into what users feel?
  2. Several servers each report their own p99. How do you get one p99 for the whole fleet?
  3. The p99 of a low-traffic endpoint jumps around. Do you trust it, and what would you use instead?

A strong answer covers

  • p50 is the typical request. p99 is the latency that only one request in a hundred exceeds.
  • A few very slow calls leave the median where it was and raise the average by little.
  • A page needs all of its calls to be fast, so the chance of meeting the tail grows with the number of calls.
  • Read a high percentile next to the median, and look at the shape of the distribution.
  • Split by server to see where the tail comes from.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Research paper

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.