Distributed Tracing — Easy Notes in 20 Points

20 revision points with ASCII diagrams covering tracing concepts, sampling, context propagation, and production lessons.

Distributed Tracing — Easy Notes in 20 Points

1. Logging vs Distributed Tracing

Main difference:

LOGGING
Service A ──> log
Service B ──> log
Service C ──> log

Question:
"What happened?"
TRACING

Request
   |
   v
Gateway
   |
   +----> Auth
   |
   +----> Order
             |
             +----> Inventory
             |
             +----> Payment

Question:
"How did the request travel,
in what order, and where was time spent?"
  • Logs = events
  • Traces = causality + timing
  • A trace_id only groups related events.
  • True tracing additionally provides the relationship between operations and their timing.

2. Trace and Span — The Basic Building Blocks

A Trace represents the complete journey of one request.

A Span represents one meaningful unit of work within that request.

Trace T100
   |
   +--> Span S1 : Gateway request
           |
           +--> Span S2 : Auth call
           |
           +--> Span S3 : Order call
                    |
                    +--> Span S4 : Inventory
                    |
                    +--> Span S5 : Payment

Think:

Trace = WHOLE JOURNEY
Span  = ONE STEP OF THE JOURNEY

A span normally has:

span_id
trace_id
parent_id
start_time
end_time
duration
status
attributes/tags

The primary tracing object is Trace → Spans → parent-child tree/DAG, not a correlated log line.

3. Choosing the Right Span Granularity

A span should represent a meaningful unit of work, usually around a boundary.

Good candidates:

HTTP request
DB call
Cache call
Queue publish
Queue consume
External API call
Important business operation

Avoid tracing every tiny internal method:

Service
 |
 +-- methodA
 |    +-- methodB
 |         +-- methodC
 |              +-- methodD
 |                   ...

Too coarse:

Order Service = 700 ms

You don't know why it took 700 ms.

Too fine:

Request
 ├─ method1
 ├─ method2
 ├─ method3
 ├─ method4
 ├─ method5
 ├─ ... 80+ spans

Now you get noise, higher ingestion/storage cost, and difficult-to-read traces.

Rule: Trace meaningful boundaries, not every line of code.

4. Parent-Child Relationship Is the Heart of Tracing

This is where tracing becomes more powerful than simply using trace_id.

Gateway S1
   |
   +----> Auth S2
   |
   +----> Order S3
             |
             +----> Payment S4
             |
             +----> Inventory S5

This tells us:

  • Who called whom?
  • Which operation happened inside which?
  • What was the causal relationship?
  • Which operations were siblings?
  • Which operations ran in parallel?

Compare that with:

Log A --> trace_id=T100
Log B --> trace_id=T100
Log C --> trace_id=T100

We know they belong to the same request, but not their structure.

trace_id
   = grouping

parent_id + span_id
   = causal structure

5. Why trace_id Alone Is Not Enough

Suppose:

Span A --> trace_id = T100
Span B --> trace_id = T100
Span C --> trace_id = T100

You can say: These belong to the same trace.

But you cannot determine:

A called B?
B called C?
A and B ran in parallel?
B caused C?
Which failure caused another failure?

Correct structure:

Trace T100
    |
    A
   / \
  B   C
      |
      D

trace_id tells you which request; span hierarchy tells you how the request happened.

6. Timing and the Critical Path

Tracing treats duration as a first-class concept.

Example:

Total request = 900 ms

Gateway       [----------------------] 900 ms
Auth          [--] 40 ms
Order         [----------------] 700 ms
Inventory     [----] 80 ms
Payment       [-----------] 500 ms

The important question: Where did the request spend its time?

The critical path is the sequence of work that actually determines the total request duration.

Request
   |
   +--> Step A ----> Step B ----> Step C
                              \
                               Critical Path

If you optimize something outside the critical path, total request time might not improve.

7. Why Span Durations Cannot Simply Be Added

Suppose three services run simultaneously:

          0ms                  200ms

Inventory     [----------------]
Pricing       [----------------]
Recommendation[----------------]

Each takes 200 ms. Sum: 200 + 200 + 200 = 600 ms. But actual wall-clock time: 200 ms, because they ran in parallel.

SUM OF SPANS != TOTAL REQUEST TIME

when concurrency exists. This is why the trace waterfall is so useful: it shows overlap directly.

8. Context Propagation Across Services

When a request moves between services, tracing context must move with it.

Client
  |
  v
Gateway
  |
  | trace context
  v
Order Service
  |
  | trace context
  v
Payment Service

Important context includes:

trace_id
span_id
parent context
sampling flag
trace state (when applicable)

Example flow:

Gateway
Trace = T100
Span  = S1

        |
        | context
        v

Order
Trace = T100
Span  = S2
Parent = S1

        |
        | context
        v

Payment
Trace = T100
Span  = S3
Parent = S2

If you forward only trace_id, the trace may look connected but the parent-child structure is lost.

9. Async / Queue Boundaries Need Special Handling

Synchronous flow — easy to model with parent-child relationships:

API
 |
 v
Service A
 |
 v
Service B

But asynchronous flow:

API Span
   |
   v
Publish Message
   |
   v
+---------+
|  Queue  |
+---------+
   |
   | 40 seconds later
   v
Worker Span

If the worker runs much later, treating it as one continuous parent-child can make the trace look like a request was "hanging" for 40 seconds. Often a trace link is more appropriate:

Trace A
  |
  +--> Publish Span
          |
          +---- linked to ----> Worker Trace
                                  |
                                  +--> Processing Span

Async boundaries must be deliberately designed; don't assume normal request propagation automatically solves them.

10. Fan-Out and Concurrency

Tracing naturally represents fan-out:

                 Gateway
                    |
          +---------+---------+
          |         |         |
          v         v         v
       Inventory  Pricing  Recommendation

Waterfall view:

Gateway        [-------------------------]

Inventory         [------]
Pricing           [----------]
Recommendation    [----------------]

If Recommendation = 800 ms and all run in parallel, parent ≈ 800 ms. Optimizing the 100 ms branches won't significantly help until the 800 ms branch is fixed.

Tracing exposes this optimization trap visually.

11. Error Semantics Inside the Trace

Tracing places errors directly into the request structure.

Trace T100
   |
   +--> Gateway       OK
   |
   +--> Order         OK
   |
   +--> Payment       ERROR / TIMEOUT
   |
   +--> Inventory     CANCELLED

This is much more useful than scattered logs, because the trace shows where the failure happened and what happened around it.

12. Use Structured Error Status

A span should have structured status such as:

OK
ERROR
TIMEOUT
CANCELLED

Why? Tracing systems can then query:

Show all traces
where span.status = ERROR

A free-text message like "Payment was kinda slow and failed" is much harder to use for dashboards, filtering, alerting, and SLO analysis.

Also, parent OK + child ERROR is not automatically a bug:

Payment
   |
   +--> ERROR
        |
        v
     Retry
        |
        v
     SUCCESS

Parent = OK

The parent may have successfully handled the child failure through retry/fallback.

13. Sampling Is Critical in Tracing

Tracing creates huge amounts of data:

100,000 requests/sec
        |
        | 20 spans/request
        v
2,000,000 spans/sec

Keeping everything can become extremely expensive. Common approaches:

Sampling
   |
   +--> 1% of normal traffic
   |
   +--> 100% errors
   |
   +--> Slow traces
   |
   +--> Important / priority traffic

Sampling is more central to tracing than ordinary log sampling because one request can produce many spans across many services.

14. Head Sampling vs Tail Sampling

Head sampling:

Request starts
     |
     v
Decide KEEP/DROP
     |
     v
Request continues

Advantages: Simple, cheap, low infrastructure complexity. Problem: if a request is dropped early and later becomes an error, it's already lost.

Tail sampling:

Request
   |
   v
Collect/buffer
   |
   v
See outcome
   |
   +--> ERROR? KEEP
   |
   +--> SLOW? KEEP
   |
   +--> NORMAL? Maybe drop

Tail sampling is more expensive/complex but can preserve the traces you most care about.

15. Sampling Must Preserve Whole-Trace Coherence

Bad approach:

Root Span  ---> DROPPED
Child Span ---> KEPT
Child Span ---> DROPPED

Result: an orphan child span that's often not useful.

Better:

             Trace
               |
       +-------+-------+
       |               |
      KEEP            KEEP
       |               |
     Span A           Span B

The sampling decision should normally preserve the whole trace. A coherent complete trace is generally more valuable than disconnected span fragments.

16. Logs, Traces and Metrics Have Different Jobs

             Observability
                  |
       +----------+----------+
       |          |          |
      Logs      Traces     Metrics
       |          |          |
    DETAILS     JOURNEY     TRENDS
       |          |          |
     "WHY?"     "WHERE?"    "HOW MUCH?"

Logs — Detailed local events, internal service state, full-text search, individual event investigation.

Traces — Cross-service request journey, latency decomposition, critical-path analysis, finding which component is slow.

Metrics — Aggregate behavior, rates, counts, latency distributions, system health/SLOs.

Key workflow:

TRACE FIRST → LOCATE THE PROBLEM
       |
       v
LOGS SECOND → UNDERSTAND THE PROBLEM

17. Always Link Logs and Traces

Trace
  |
  +--> Slow Payment Span
           |
           | trace_id + span_id
           v
       Exact Logs
           |
           v
      Root Cause

Logs should carry trace_id and span_id. Then you can move:

Trace → Span → Logs

and also:

Log → trace_id → Full Trace

Don't think Logs VS Traces — think Logs + Traces. They answer different questions and work best together.

18. Production Deployment Lessons

80+ microservices
      |
      +--> ~35 instrumented
               |
               +--> OpenTelemetry
               +--> Collector
               +--> 5% head sampling
               +--> 100% errors + p95+ tail sampling
               +--> 6–9 avg span depth
               +--> 24h hot retention

Example business flows:

MF Redemption
Gateway → Order → NAV → Payment → Ledger → Notification

IPO Allotment
Webhook → Allotment → Notification

Demat Account
Onboarding → NSDL → KYC → Demat

Insurance
Policy Gateway → Adapter → Sync

Loan Against MF
Loan → Valuation → Lien → Disbursal

Important production lesson: External boundaries are often where trace coverage becomes incomplete, especially when internal retry/backoff logic isn't instrumented.

19. Real Production Problems and Their Fixes

Four important failure patterns:

A. Async trace broken

Publish Trace
     X
     |
     X   Worker Trace

Fix: propagate trace context via message headers + use a trace link.

B. Too many spans

1 request
   |
   +--> 80+ spans

Fix: boundary spans only → network, DB/cache, external call, key business step.

C. Important error wasn't sampled

5% head sample --> customer error --> TRACE LOST

Fix: guarantee retention of error paths through tail sampling.

D. Retry hidden inside one span

Payment Span = 3 seconds
Actually:
Attempt 1 → fail
Attempt 2 → fail
Attempt 3 → success

Fix: model retries as child spans with an attempt-number tag.

20. Final Mental Model — Remember This for Interviews

                 ONE REQUEST
                      |
                      v
               ROOT SPAN
                      |
                      v
          +-----------+-----------+
          |                       |
       Child Span             Child Span
          |                       |
          v                       v
     Downstream A            Downstream B
          |                       |
          +-----------+-----------+
                      |
                      v
             CONTEXT PROPAGATION
                      |
        +-------------+-------------+
        |                           |
      SYNC                        ASYNC
        |                           |
   Parent/Child                Trace Link
        |                           |
        +-------------+-------------+
                      |
                      v
                 TIMING
                      |
                      v
              CRITICAL PATH
                      |
                      v
                  SAMPLING
                      |
             +--------+--------+
             |                 |
           HEAD               TAIL
             |                 |
          cheap              smarter
          but blind         outcome-aware
                      |
                      v
              OBSERVABILITY
                      |
        +-------------+-------------+
        |                           |
      TRACE                       LOGS
     "WHERE?"                     "WHY?"

20-point memory shortcut

1.  Logs = events
2.  Traces = request journeys
3.  Trace = collection of spans
4.  Span = meaningful timed work
5.  Span hierarchy = causality
6.  trace_id alone ≠ tracing
7.  parent_id connects the journey
8.  Duration is first-class
9.  Critical path determines latency
10. Parallel spans explain concurrency
11. Propagate complete trace context
12. Async boundaries need deliberate design
13. Fan-out = sibling spans
14. Errors need structured status
15. Sampling is essential at scale
16. Tail sampling preserves important traces
17. Sample whole traces coherently
18. Trace = "where", logs = "why"
19. Link logs ↔ traces
20. Mature observability = collect + watch + act

One interview sentence to memorize:

"Distributed tracing turns one request into a causally connected, timed span tree across services, allowing us to see the execution path, identify the critical path, understand failures and concurrency, and then use logs to investigate the exact root cause."

Distributed Tracing — Easy Notes in 20 Points