Each page a user opens makes 30 calls, spread over three servers. Every call comes back in about 20 ms, and the dashboard shows that one typical number.
call
fast answer
Arrow keys to move · 1 to 4 to answer
In short
p50 vs p99: how it works and how it breaks
The median, or p50, is the latency of the typical request: half are faster. The 99th percentile, p99, is the latency that only one request in a hundred exceeds. They answer different questions. If one server in three is slow for a small share of its calls, one call in fifty can take ten times longer while the median stays where it was and the average rises by a few milliseconds. A dashboard showing only those two stays green. Users do not experience single calls, though. A page that makes thirty calls needs all thirty to be fast, so a slow two percent of calls reaches almost half of the page loads. Read a high percentile next to the median, look at the shape of the distribution and not one number, and split it by server to see where the tail comes from.
You cannot average percentiles
The mean of three servers' p99 values is not the fleet's p99: two servers at 25 ms and one at 200 ms average to a number no request experienced. Percentiles computed inside each process cannot be combined afterwards. Export histogram buckets, which are plain counters, sum each bucket across servers, and compute the percentile from the sums. The Prometheus docs give this as the main reason to prefer histograms when a service has several instances.
Bucket edges limit what you can read
A percentile taken from a histogram is estimated inside the bucket it falls in, so it is only as precise as that bucket is narrow. With edges at 100 ms and 500 ms, a p99 of 110 ms and one of 450 ms can report much the same value. Put a bucket edge exactly on your latency target. The share of requests under that edge is then an exact count, and often a better alerting signal than an estimated percentile.
Fan-out turns a rare tail into the common case
If a user action needs n calls and each has a 1% chance of being slow, the action is slow with probability 1 − 0.99^n: about 26% at 30 calls and 63% at 100. A backend's p99 is closer to the typical page experience than to its worst case. Set backend targets at higher percentiles than the page target, and also measure latency where the user's request enters, across the whole fan-out.
Load tests that hide their own tail
A load generator that sends the next request only after the previous one returns slows down whenever the server stalls. The requests that should have been sent during the stall are never sent or measured, so the reported p99 looks far better than what users would see. This is called coordinated omission. Send at a fixed rate that does not depend on responses, and time each request from when it was scheduled to go.
Check how many samples sit behind the number
A p99 over 200 requests is set by the two slowest, so on a quiet service or a short window it jumps around and says little. Before acting on a tail percentile, check the request count in the same window. For low-traffic endpoints use a longer window, or count requests over the target directly. Keep the per-server view too: one slow instance out of many can vanish in a fleet-wide percentile.