Skip to content

Observability · 5 of 6

Step 1 · The pattern

Three services, one log store

A request passes through three services. Each one writes its own log line, and all lines land in one log store, mixed with the lines of every other request in flight. Nothing on a line says which request it belongs to.

  • request
  • log line
  • served

In short

Log correlation via a trace ID: how it works and how it breaks

Each service writes its own log lines. When one request fails, the lines that explain it are spread over several services and buried among lines from every other request handled in the same moment. Matching by timestamp fails: the same second holds many requests, and the line that matters may have been written seconds later, when a timeout finally expired. You end up blaming the wrong service. The fix is a trace ID. The first service to see a request gives it a unique ID. Every service passes the ID to the next in a header, and writes it on every log line. One search on that ID then returns exactly that request's lines, from all services, in order, whenever they were written. It only works if every hop passes the ID on: one hop that drops it cuts the story in two.

How this comes up in interviews

"One checkout fails in a system of several services that each write their own logs. How do you find every log line that belongs to that request?"

Then they ask

  1. A search by trace ID returns lines from the API but none from the queue worker that handled the order. Why?
  2. The services' clocks agree. Why is matching lines by timestamp still not good enough?
  3. A public client sends its own trace ID in a header. Do you trust it, and what may it be used for?

A strong answer covers

  • Timestamps fail: the same second holds many requests, and the line that matters may be written seconds later.
  • The first service to see a request gives it a unique trace ID.
  • Every service passes the ID to the next in a header and writes it on every log line.
  • One search on the ID returns that request's lines from all services, whenever they were written.
  • One hop that drops the ID cuts the story in two. Queues and background jobs are the usual place.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Standard

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.