What Is Distributed Tracing? A Guide for Product Engineers

Distributed tracing is the practice of following one request across every service it touches. Each service records its work as a span and tags it with the same trace ID, passed along in request headers. The result is a single timeline that shows the full path of the request, how long each step took, and where it failed.

In a single application, a stack trace often tells you enough. Microservices change that. One checkout request can cross five services, each owned by a different team with its own logs.

Without a shared ID, connecting those logs during an incident means matching timestamps by hand. Tracing replaces that guesswork with one view of the whole request.

Distributed tracing works by starting a trace when a request enters your system, recording each unit of work as a span, and passing trace context along with every call so downstream spans join the same trace.

Walk through one request

Take a hypothetical ecommerce checkout:

1. A shopper taps "Place order" in your mobile app. The API gateway receives the request, creates a trace ID, and opens the first span. Because it has no parent, it's called the root span.
2. The gateway calls the order service and sends the trace context in a header. The order service opens a child span under the gateway's span.
3. The order service calls inventory and payments in parallel. Each opens its own child span.
4. The payment service calls a third-party payment provider, which takes several seconds to respond.
5. The response returns to the app. Your tracing backend now holds every span, which it draws as a waterfall.

Read top to bottom, the waterfall shows the slow payment span right away. You no longer have to ask which of five teams owns the problem.

Traces, spans, and attributes

A trace is the whole tree. A span is one operation inside it: a database query, an HTTP call, a function you chose to instrument. Each span stores a name, start and end times, its parent span's ID, a status, and attributes, which are key-value pairs such as http.status_code or order.id.

Attributes make traces searchable. A span that records the region, app version, or customer plan lets you filter thousands of traces down to the few that matter.

Context propagation

Context propagation is how each service learns which trace it belongs to. OpenTelemetry's definition of traces and spans puts it plainly:

"Context Propagation is the core concept that enables Distributed Tracing."

In practice, each outgoing call carries a header with the trace ID and the ID of the calling span. The W3C Trace Context standard, a W3C Recommendation since 2021, defines that header as traceparent, with fields for version, trace ID, parent ID, and trace flags. A shared format means a span from one vendor's instrumentation can link to a span from another's, instead of every tool inventing its own header.

Distributed tracing shows one request's path across services, while logs show what happened inside one service, metrics show overall health, and product events show who was affected and what it cost.

The OpenTelemetry docs describe traces as a kind of structured log with correlation and hierarchy built in. A log line tells you one service's side of the story, and a trace puts every service's side in order.

Product events come from event tracking, and they sit in a different layer. They won't show you the slow span. In a hypothetical case, they'd show you that checkout completions dropped right after a deploy.

OpenTelemetry tracing is the open, vendor-neutral way to instrument code and emit traces, so you can send the same data to different backends without rewriting your instrumentation when you switch tools.

OpenTelemetry, often shortened to OTel, standardizes what a span contains and how context moves between services. That's why it has become the vendor-neutral default for instrumenting microservices, whatever backend you choose.

The model is older than OTel. Google's 2010 research paper Dapper, a large-scale distributed systems tracing infrastructure described requests as trace trees made of spans, the model modern tracers still use.

OTel handles collection. It doesn't store or display traces. For that, you pair it with a backend such as Jaeger or Zipkin, two widely used open-source tracing systems that store, search, and visualize trace data.

A common setup is to instrument with OpenTelemetry and send traces either to an open-source backend like Jaeger or Zipkin, or to a commercial application performance monitoring (APM) platform like Datadog, Dynatrace, or New Relic.

Open-source backends

Jaeger and Zipkin give you trace search, waterfall views, and service dependency graphs without a license fee. You keep full control of your data. The tradeoff is operations: your team runs the storage, scaling, and retention.

Commercial APM platforms

Datadog, Dynatrace, and New Relic manage storage for you. They also put traces next to logs, metrics, alerting, and infrastructure data in one place. That correlation shortens incident response. Cost tends to grow with the volume of data you send, which shapes how much you can afford to keep.

Where product analytics fits

Neither type tells you which customers a slow span hit. Product analytics sits next to your tracing backend for that. For broader app monitoring advice, see our tips for app performance tracking.

The main challenges of distributed tracing are sampling and cost, broken context propagation, thin frontend coverage, and the overhead of instrumentation.

The tooling problem runs alongside a practice problem. Zhou, Peng, Xie, and colleagues paired a survey of industry practitioners with a benchmark microservice system to study how teams debug faults. Their industrial study of debugging microservice systems found that tracing and visualization help, and it called for smarter ways to analyze traces.

Sampling and cost

Keeping every span from every request gets expensive fast, so teams sample. Head-based sampling decides at the start of a request. Tail-based sampling decides after the trace completes, which helps you keep the errors but holds more data in memory. The W3C spec also warns that if services record different subsets of requests, the result can be broken traces.

Broken context propagation

One service that drops the header splits a trace into fragments. Common culprits include older libraries, message queues, and third-party services you don't control. Shared standards help, but someone still has to check that context survives every hop.

Frontend gaps

Many traces start at the API gateway, which means they miss the part users see. The tap, the slow render, and the shopper's frustrated retry never appear. Session replay and frontend error capture fill part of that gap.

Overhead and instrumentation effort

Each span adds a little processing and network cost. Automatic instrumentation covers common frameworks, but the attributes that make traces useful to product teams, such as plan, cohort, or experiment variant, usually take custom work.

You trace an AI agent the same way you trace microservices: the agent run is the trace, and each model call, tool call, and retrieval is a span with its own timing, inputs, and status.

Take a hypothetical fintech support assistant. A customer asks why a transfer is late. The root span covers the full agent run.

Below it, child spans cover the retrieval of policy documents, a model call that plans the answer, a tool call to the transactions API, and a final model call that writes the reply. If context propagates into the transactions API, its backend spans join the same trace.

The big difference is what "failed" means. A span can return a 200 status while the answer it fed is wrong. That's why agent teams pair traces with evaluations and trace analysis, inspecting runs one by one and in aggregate to find failure patterns.

If the concept is new to you, start with what an AI trace is. LLM observability covers the wider monitoring picture for AI apps.

Connect traces to user impact by joining each slow or failed span to the sessions, funnel steps, and cohorts it touched, then fix the problems that cost the most users first.

Here's a workflow you can start this sprint:

1. Record the trace ID where users are. Add it as a property on the product events and frontend errors tied to each request, so engineering and product data share a key.
2. Find the funnel step. When a span starts failing, check which step of the journey it sits behind and whether funnel drop-off moved at the same time.
3. Size it by cohort. Use cohort analysis to see whether the failure hits new users, a single platform, or your highest-value accounts.
4. Watch the affected sessions. Replays show what the user saw while the backend struggled.
5. Rank fixes by users and revenue affected, not by error count alone.

That last step changes the backlog. Picture two hypothetical errors with the same rate: one on an admin settings page, one at payment for first-time buyers. The traces look equally urgent. The user impact doesn't.

Amplitude covers the user-impact half. Session Replay captures network and console errors on the replay timeline, and you can open replays from Amplitude charts, such as funnels, or from User Look-Up. For AI features, Amplitude Agent Analytics ties agent traces and evals to conversion, retention, and revenue, so you can see which failures cost you users. Amplitude works alongside your observability stack, not instead of it: keep Jaeger or your APM platform for storing and searching backend spans.