# Why does an instance lose parent-child span correlation across LLM calls, retrievals, and agent handoffs **Date of Record:** 2026 November 15 **Scope:** Applies to distributed tracing of GenAI workloads where a single logical operation (a user request, an agent turn) is composed of heterogeneous span-emitting components — LLM inference clients, vector retrievers, tool executors, and sub-agent handoffs — each potentially running in a different process, container, or trust domain. Does not apply to single-process, single-threaded tracing where the span context is passed by direct object reference and never crosses a serialization boundary. ## Core Distinction The failure is not "missing traces" and it is not "broken spans." It is **context propagation loss at a heterogeneous boundary**: the trace exists, every span exists, every span has a valid `trace_id` — but the `parent_span_id` on one or more spans points to a span that the collector never received, or points to `null` where a parent should exist. The result is a *disconnected trace*: a tree that renders as a forest. This is distinct from dropped spans (which produce gaps but preserve tree structure among survivors) and distinct from clock skew (which misorders spans but preserves parentage). The invariant that breaks is: **for every non-root span, the parent span must be present in the same trace and must have a `start_time` strictly less than the child's `end_time`.** The non-obvious part is *where* it breaks. It almost never breaks at the LLM call itself. It breaks at the seams: the retrieval client that constructs a new HTTP request without injecting the W3C `traceparent` header; the agent handoff that serializes task state to JSON and reconstructs it in a worker that never saw the ambient context; the async task spawned via `asyncio.create_task` that inherits the event loop but not the context variable holding the current span. ## What Was Observed In a three-tier agent pipeline (orchestrator → retrieval sub-agent → LLM synthesis), traces arrived with the orchestrator root and the final LLM span both present and correctly linked by `trace_id`, but with the retrieval spans forming an isolated subtree whose root had `parent_span_id = null`. The collector's trace assembly reported the trace as complete (all spans present) yet the tree view showed two roots. Instrumentation inspection revealed the mechanism: the retrieval sub-agent was dispatched across a process boundary using a task queue. The orchestrator's span context lived in an OpenTelemetry `contextvars.ContextVar`. The task payload was serialized with `json.dumps` of a plain dict containing the query and metadata — **the span context was not in that dict**. The worker process, on deserialization, called `tracer.start_as_current_span("retrieval")` with no active context, so it created a new root. The `trace_id` happened to match only because the worker was also configured to reuse a propagated `trace_id` from a header that the queue transport set independently — producing the deceptive appearance of a shared trace with a broken tree. A second, subtler observation: in the async path, `contextvars` propagate into `asyncio.create_task` (the task copies the current context at creation), but **not** into `loop.run_in_executor` / `ThreadPoolExecutor` submissions, and **not** across `await` boundaries where an intermediate library explicitly resets context (some HTTP client connection pools do this). Spans created inside the executor thread became roots even though the calling coroutine held a valid current span. ## Condition of Validity The parent-child invariant holds **only when the span context is carried across every boundary by an explicit, serialized carrier** — the W3C `traceparent`/`tracestate` headers for HTTP, a message attribute for the queue, an explicit field in the task payload for in-process handoffs. It is **not** valid to rely on: - ambient `contextvars` crossing a process boundary (they do not), - ambient `contextvars` crossing a thread-pool boundary (they do not), - the transport auto-populating a `trace_id` (it may set the ID without setting the parent), - a library's "auto-instrumentation" for a protocol it does not actually support (silent no-op). The invariant is also invalidated by **span-context reuse across retries**: if a retry reuses the original span object rather than creating a child, and the original span's parent was from a prior request, the retry's parentage is wrong even though both spans exist. ## Decision Boundary When a trace renders as a forest, the decision is: **do not attempt to repair parentage post-hoc by heuristic stitching** (matching on timestamps or `trace_id` proximity). Heuristic stitching produces plausible but false trees and destroys the auditability that motivated tracing. Instead: 1. Treat a forest as a **propagation defect**, not a display defect. Locate the boundary where `parent_span_id` became null. 2. If the boundary is a process/queue hop, the fix is to add the carrier to the payload schema and inject/extract at the hop — a schema change, not a tracing-config change. 3. If the boundary is a thread-pool or executor hop, the fix is explicit context capture at submission (`contextvars.copy_context().run(...)`) — a code change at the call site. 4. If the boundary is a retry, the fix is to create a fresh child span per attempt with an `attempt` attribute, never reuse. The boundary between "acceptable" and "unacceptable" is whether the missing parent is *recoverable from the carrier*. If the carrier never held the context, the trace is irreparably incomplete and must be flagged as such, not silently stitched. ## Operational Effect Downstream consumers of these traces — cost attribution, latency decomposition, agent-decision audits — silently degrade. Cost attribution that sums child-span token counts under a parent undercounts by exactly the fraction of spans that became roots. Latency decomposition that computes `parent.duration - sum(child.duration)` attributes the entire retrieval latency to "orchestrator overhead," inflating the orchestrator's apparent cost and hiding the retriever as a bottleneck. An agent reading these traces to make a routing decision (e.g., "the retriever is slow, bypass it") will make the wrong decision with high confidence, because the data looks complete. The operational rule: **a trace with more than one root is a data-quality incident, and any metric derived from it must be quarantined until the propagation boundary is repaired.** The following guard enforces this at trace-assembly time. ```python from dataclasses import dataclass, field from typing import Optional class SpanCorrelationError(RuntimeError): """Raised when a trace's parent-child invariant is violated.""" pass @dataclass class Span: span_id: str trace_id: str parent_span_id: Optional[str] name: str start_ns: int end_ns: int @dataclass class Trace: trace_id: str spans: list[Span] = field(default_factory=list) def validate_trace_invariant(trace: Trace) -> None: """ Enforce: every non-root span has a parent present in the same trace, and parent.start_ns < child.end_ns. A trace with >1 root is a propagation defect, not a display artifact. """ if not trace.spans: raise SpanCorrelationError(f"trace {trace.trace_id} has no spans") by_id = {s.span_id: s for s in trace.spans} roots = [s for s in trace.spans if s.parent_span_id is None] if len(roots) > 1: root_names = ", ".join(sorted(r.name for r in roots)) raise SpanCorrelationError( f"trace {trace.trace_id} has {len(roots)} roots ({root_names}); " f"span context was lost at a heterogeneous boundary " f"(process hop / thread-pool hop / retry reuse). " f"Do not heuristic-stitch; locate the missing carrier." ) for span in trace.spans: if span.parent_span_id is None: continue parent = by_id.get(span.parent_span_id) if parent is None: raise SpanCorrelationError( f"span {span.span_id} ({span.name}) references parent " f"{span.parent_span_id} absent from trace {trace.trace_id}; " f"collector dropped the parent or carrier was never populated." ) if parent.start_ns >= span.end_ns: raise SpanCorrelationError( f"span {span.span_id} ({span.name}) ends at {span.end_ns} " f"before parent {parent.span_id} starts at {parent.start_ns}; " f"clock domain mismatch across the boundary." ) def assemble_and_quarantine(trace: Trace) -> Trace: """Validate before any metric derives from the trace.""" try: validate_trace_invariant(trace) except SpanCorrelationError: # Quarantine: do not emit cost/latency metrics for this trace. trace.spans = [s for s in trace.spans if s.parent_span_id is None] raise return trace ```