Skip to content

Observability · 4 of 6

Step 1 · The pattern

One request, many calls

A user loads an order. The gateway passes the request to the order service, which checks the token with auth and then reads the database. The user sees one number: how long the gateway took.

  • request
  • fast reply

In short

Trace waterfall: spot the slow span: how it works and how it breaks

One user request becomes many calls between services. A trace records each call as a span: a name, a start time, a duration and a parent. Drawn under each other, the spans form a waterfall. A parent span is open for as long as its children run, so the bar at the top is always the widest. That bar tells you the request was slow. It does not tell you where. To find out, read downwards: from the widest bar go to its widest child, and repeat until you reach a span with no slow children. That span is where the time goes. Two shapes are worth knowing. One long bar is a single slow operation, such as a query without an index. A long row of short bars one after another is many small calls made in sequence, and no single one of them looks slow.

How this comes up in interviews

"A request that crosses several services has become many times slower, and the only number on the dashboard is gateway time. How do you find where the time goes?"

Then they ask

  1. Under one parent you see dozens of short spans, one after another, and none is slow. What is it, and how do you fix it?
  2. Part of a parent's bar is covered by no child span. What could that time be?
  3. You keep only a sample of traces. How do you make sure the slow request you care about was kept?

A strong answer covers

  • A trace records each call as a span: a name, a start time, a duration and a parent.
  • A parent stays open while its children run, so the top bar is always the widest. It says the request was slow, not where.
  • Follow the widest bar to its widest child, and repeat until you reach a span with no slow children.
  • One long bar is a single slow operation. A long row of short bars is many small calls made in sequence.
  • Scaling the service at the top changes nothing if it was only waiting.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Research paper

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.