Skip to main content

Tracing the Critical Path End-to-End

What This Concept Is​

A distributed trace is a tree of timed operations (spans), all sharing one trace ID, that together describe a single user request as it crosses processes, services, queues, and dependencies. The trace is the only observability signal that answers "where did the time go?"

Tracing in practice means three decisions:

  • what to instrument -- which entry points, which hops, which internal boundaries
  • what to name -- span names should be low-cardinality verbs (GET /webhook, db.insert events, kafka.publish), not one-off strings with IDs baked in; OpenTelemetry calls this the "span name" convention and requires it be a low-cardinality representation of the operation.
  • what to attribute -- attributes on spans carry context (user, tenant, route, size); high-cardinality items go here, following OpenTelemetry semantic conventions (http.route, messaging.destination.name, db.system).

A subtle point often missed: OpenTelemetry has probabilistic sampling (head-based) and tail-based sampling, and a well-run pipeline combines them. Head sampling at the SDK is cheap but blind; tail sampling in the collector keeps every trace that errored or exceeded a latency threshold, so you retain the traces that matter without keeping them all.

For a capstone, the critical path is the one user journey your SLO is built on. Trace that journey, fully. Do not try to trace everything. A partial trace of the SLO path is infinitely more useful than a complete trace of the health-check endpoint.

Traces also unify your signals: logs and metrics correlated by trace_id become cross-service breadcrumbs, and OpenTelemetry exemplars attach a trace ID to a specific metric point so that "the 99th percentile went up" becomes "the 99th percentile went up and here is the trace that caused it."

Why It Matters Here (In the Capstone)​

Traces are the only observability signal that answers "where did the time go?" The dashboard tells you p95 is 800 ms; only the trace tells you 600 ms of it was spent in an external API call, or in a serial loop that should have been parallel.

Runbook entries that say "open the trace for trace_id = <foo>" are worth ten entries that say "check the logs in service X, then service Y, then..." Concept 13 runbooks assume you have this capability. The "top 10 slow traces" panel in concept 5 assumes these traces are retained -- which depends on the sampling policy below.

Concrete Example(s) -- from a real capstone​

Critical path for the webhook-handler capstone: POST /webhook -> verify signature -> rate-limit check -> publish to queue -> queue consumer -> write to DB -> notify downstream.

A well-formed trace looks like this (indentation = parent/child):

trace_id = t-a8f3...
span  POST /webhook                 350 ms   service=api     attrs: http.route=/webhook, provider=stripe
  span  verify_signature              3 ms   service=api     attrs: algo=hmac-sha256
  span  rate_limit.check             12 ms   service=api     attrs: bucket=stripe, allowed=true
  span  kafka.publish                 8 ms   service=api     attrs: messaging.destination.name=events.incoming, partition=3
  (async, same trace via traceparent propagation)
  span  consumer.handle events      290 ms   service=worker  attrs: messaging.destination.name=events.incoming
    span  db.insert events           14 ms   service=worker  attrs: db.system=postgresql, db.sql.table=events
    span  http.post downstream api  260 ms   service=worker  attrs: server.address=notify.svc, http.response.status_code=200

Four lessons from this trace:

  1. The API endpoint is not slow. Everything synchronous in /webhook completed in ~25 ms.
  2. The total experience is slow because the downstream notification takes 260 ms.
  3. consumer.handle events is the place to optimize, not verify_signature.
  4. The attributes let you filter: in the dashboard, traces where provider=stripe could be slow while other providers are fine.

Without the trace, you would see a 350-ms p95 on one graph, a high consumer latency on another, and have to guess which caused which. With the trace, it is one screen.

Sampling policy to test before release:

Choose a pipeline based on the evidence you need. One controlled experiment sends all spans in the selected critical-path population to the Collector, then applies an outcome-aware tail policy with a bounded normal-traffic sample. Provision and observe buffering, late spans and overload behavior; this is not a guarantee of complete capture.

If you instead head-sample 2% at the SDK, the tail sampler can see only that retained population. It cannot recover the other 98%, even if those traces failed. For 1,000 independently sampled error traces, expected retention before further filtering is 20, not 1,000. The observed count is random and transport loss can lower it further.

Generate known normal, slow and error requests and compare sent, received and stored counts by class. Record upstream sampling, late spans and drops. Choose the release policy from those results and the resource budget; do not infer an overall storage percentage from configuration alone. See OpenTelemetry sampling.

Common Confusion / Misconceptions​

"We already have logs -- traces are redundant." Logs answer "what happened and why?" Traces answer "where did the time go and in what order?" They are complementary. In a trace, parent-child relationships and durations are the point; logs cannot express those cleanly.

"One sampling rate fits every workload." Full capture may be feasible for a bounded low-volume experiment; a high-volume service may need sampling. Measure volume, retention and diagnostic coverage rather than prescribing one percentage for every capstone.

"Span names with the user ID in them are easier to search." They are also a cardinality explosion. Span names should be low-cardinality (db.insert events). Put the user ID on an attribute (user.id = "u-123") and use the attribute for filtering.

"Tracing means OpenTelemetry everywhere right now." It means instrument the critical path first. You do not need global coverage on day one. Entry point, every outgoing call on the critical path, and every async boundary -- that is the minimum.

"We'll sample 100% in dev and change nothing in prod." Dev and prod have different distributions; the tail in prod is where the interesting traces live, and a 100% dev sample rate teaches you nothing about what tail sampling will look like. Match your prod sampling policy in a staging environment of reasonable load at least once before go-live.

"Spans under 1ms are not worth keeping." Often true for averages; dangerous for tail analysis. Aggregate trivial spans into a single internal span if you must, but never drop them outright -- the one-millisecond span that happens 50 times in a serial loop is exactly the bug a trace is built to find.

How To Use It (In Your Capstone)​

  1. Name the one critical path for your capstone (the SLO path).
  2. Instrument both ends: the entry point (HTTP, webhook, cron) and every outbound call (DB, HTTP, queue) on that path. Use OpenTelemetry auto-instrumentation for your framework where available -- hand-written spans only where the framework cannot see the boundary.
  3. Propagate traceparent across every hop (W3C TraceContext headers for HTTP, message attributes for queues).
  4. Use OpenTelemetry semantic conventions for span names and attributes (http.*, db.*, messaging.*) instead of inventing your own.
  5. State the sampling population and upstream/tail decisions. Verify normal/error/slow trace counts and loss; a downstream tail sampler cannot recover traces discarded at the SDK.
  6. Pick one recent real trace per week. Walk it in your tracing UI. If any significant hop is missing, add instrumentation. Repeat until the trace explains a full request end to end.
  7. Link trace URLs into runbooks and log lines (the same trace_id field from concept 4) so "the trace" is a click, not a search.

A 30-minute "trace walk" you should run weekly:

  1. Pick a random trace from the last hour that is at or near your p95 latency line.
  2. Read the spans top-to-bottom. Sum the durations. Verify they roughly equal the root span's duration (gaps mean uninstrumented work).
  3. Identify the single slowest child span. Ask: is that hop's slowness explained by its own attributes (e.g., a large payload, a cache miss) or external (a slow dependency)?
  4. File one follow-up ticket -- either an instrumentation gap or an optimisation opportunity. This cadence is how tracing coverage improves without a "tracing project."

See also (integrative)​

Check Yourself​

  1. Why should a span name be low-cardinality but span attributes can be high-cardinality?
  2. What is the argument for always keeping traces of 5xx responses, regardless of your sample rate?
  3. How does a trace help you decide whether a p95 regression is your code or a downstream dependency?
  4. What does W3C TraceContext (the traceparent header) guarantee that a random correlation ID would not?
  5. When does combining head and tail sampling lose error coverage, and how would you measure that loss?
  6. If you could only instrument one span for your capstone today, which would it be and why?

Mini Drill or Application (Capstone-scoped)​

  1. In 45 minutes: instrument one endpoint in your capstone with OpenTelemetry (or the native tracer for your stack).
  2. Ensure at least three downstream calls show up as child spans with OpenTelemetry-compliant attribute names.
  3. Force an error (e.g., point one dependency at a bad host) and confirm the error trace is retained by the tail sampler.
  4. Reproduce one real slow request and walk the trace until you can identify the hop that consumed > 50% of the latency.
  5. Write one paragraph in library/raw/tracing.md describing: which path is instrumented, the sampling policy, and how a runbook should link to a trace (<tracing-ui>/trace/<trace_id>).

Source Backbone​

Capstone operations applies security, reliability, and distributed-systems material. These books are the source backbone for readiness review.