A request passes through three services. Each one writes its own log line, and all lines land in one log store, mixed with the lines of every other request in flight. Nothing on a line says which request it belongs to.
request
log line
served
Arrow keys to move · 1 to 4 to answer
In short
Log correlation via a trace ID: how it works and how it breaks
Each service writes its own log lines. When one request fails, the lines that explain it are spread over several services and buried among lines from every other request handled in the same moment. Matching by timestamp fails: the same second holds many requests, and the line that matters may have been written seconds later, when a timeout finally expired. You end up blaming the wrong service. The fix is a trace ID. The first service to see a request gives it a unique ID. Every service passes the ID to the next in a header, and writes it on every log line. One search on that ID then returns exactly that request's lines, from all services, in order, whenever they were written. It only works if every hop passes the ID on: one hop that drops it cuts the story in two.
What the traceparent header contains
W3C Trace Context defines one header, traceparent, with four dash-separated hex fields: version (2 characters, currently 00), trace-id (32), parent-id (16, the caller's span) and trace-flags (2, whose lowest bit means sampled). An all-zero trace-id or parent-id is invalid. The trace-id stays the same for the whole request; the parent-id changes at every hop. Log both: the trace-id finds the request, the span id tells you which call within it wrote the line.
Create it at the edge, show it to the user
Generate the ID in the first component you control, so the load balancer's and gateway's own lines carry it too. Then hand it back: print it on the error page and in the error body of your API. A support ticket that quotes the ID turns an hour of searching into one query. Decide deliberately which responses include it, since the W3C spec warns that IDs in responses can leak to cross-origin callers.
Queues and background jobs lose it first
HTTP and RPC libraries usually forward the header for you. A message on a queue, a scheduled job or a hand-rolled thread pool does not: the producer must copy the context into the message's metadata and the consumer must restore it before it logs. Audit these hops explicitly. For a batch consumer handling many messages at once, set the context per message, or every line in the batch is filed under the first message's ID.
Make the ID a field, not a piece of text
An ID inside a free-text message can only be found by scanning text. Write structured logs with the trace ID as its own field, under the same field name in every service and language, and have the logging library add it from the request context so nobody types it by hand. OpenTelemetry SDKs can attach the trace and span IDs to log records automatically. Check that your log store indexes that field.
An incoming ID is a hint, never a credential
Anyone can send a traceparent header. Validate its length and format, and use it only to correlate: never to authorise, deduplicate or rate limit. At a public entry point, consider starting a fresh trace and recording the caller's ID as an attribute; the W3C spec describes restarting a trace at secure network entry points. Otherwise a caller who always sets the sampled flag can inflate your tracing bill, or reuse one ID to tangle unrelated requests together.