# Where must an instance propagate trace context when an agent span suspends for an asynchronous tool call, and why the naive handoff silently orphans child spans **Date of Record:** 2026 November 17 **Scope:** Applies to multi-agent and tool-using LLM systems that emit OpenTelemetry-compatible spans across at least one asynchronous (deferred, queued, or callback-driven) tool invocation. Does not apply to fully synchronous in-process tool calls where the parent span's execution context remains live for the entire child lifetime. ## Core Distinction The boundary in question is not "does the trace ID survive serialization." It is: **at which instant does the *active span* (the unit of parentage) transition from the suspended agent span to the resumed child span, and who owns that transition?** A trace context (`trace_id`, `span_id`, `trace_flags`, `trace_state`) is a passive value. Propagation is trivially correct in the sense that the bytes round-trip. The failure is that *parentage is established at span creation time, not at span completion time*. When an agent span suspends to await an async tool result, the runtime's active context is popped or replaced. Any child span created after the await boundary — including the tool-execution span, retry spans, and downstream LLM calls — is parented to whatever context is active at creation, which is frequently a fresh root or a sibling, not the suspended agent span. The non-obvious invariant: **a suspended span must remain the recorded parent of children created during its suspension, even though its execution context is no longer the active one.** Passive context propagation preserves the IDs; it does not preserve the *active-parent* relationship. Those are different guarantees, and only the second one keeps the trace tree intact. ## What Was Observed In a three-level agent topology (orchestrator → planner agent → tool-calling agent → async retrieval tool), the orchestrator and planner spans appeared correctly nested. The tool-execution span and everything beneath it appeared as a **separate root trace** with a different `trace_id`, or — when a W3C `traceparent` was threaded manually — as a child of the planner span but with a **missing intermediate tool-call span**, producing an orphaned subtree whose parent span ID referenced a span that had already been exported and closed. Two concrete signatures recurred: 1. **Root-split:** the async tool worker started a new trace because the carrier injected into the job payload lacked `trace_flags` (sampled bit lost), so the worker's propagator treated it as invalid and fell back to a new root. The IDs were present but the sampled flag was zero. 2. **Orphan-by-early-close:** the tool-call span was ended when the await began (to "avoid a long-running span"), so children created by the callback referenced a closed span ID. Backends that validate parent existence at ingest drop or quarantine the subtree; backends that do not validate render it detached. Neither failure throws. Both are silent in application logs and only visible in the trace backend as split or orphaned trees. ## Condition of Validity The invariant holds — and the failure is *in scope* — only when all of the following are true: - The tool call is genuinely asynchronous: the calling agent's coroutine/thread releases control and a different execution context (worker, event-loop callback, queue consumer) runs the tool. - Child spans are created *after* the await boundary, not before it. - The trace backend validates parent-child linkage (most do, at least for sampling decisions). It does **not** apply when the "async" tool is really a synchronous call behind a future that the same context awaits inline — there the active context is never actually displaced, and naive propagation works. It also does not apply to fire-and-forget spans that are intentionally detached (e.g., audit events), which should be marked as links, not children. ## Decision Boundary The decision is not "propagate context or not." It is **where to re-establish the active parent on resume**. - If the tool worker runs in a context where the suspended agent span *can* be made active again cheaply (same process, context variable restorable from the carrier), re-activate the parent span before creating the tool-execution span, and do **not** end the agent span until its last child is created. - If the worker is in a different process or cannot restore the parent (the common case for queued tools), the correct move is to **inject the full carrier including `trace_flags` and `trace_state`**, create the tool-execution span as a child of the *recorded* parent span ID (not the active one), and accept that the agent span must remain open until the tool result returns — or, if it cannot, emit an explicit **span link** rather than a false parent reference, and mark the tool span's parent as the link target. The boundary to refuse: never end the agent span at the await point and then create children against its ID. That is the orphan generator. Either keep the parent open across the suspension, or switch to link semantics — never fabricate a parent that no longer exists. ## Operational Effect When the invariant is honored (parent kept open across suspension, or link semantics used), traces render as a single tree and latency attribution across the async boundary is correct: the tool-execution span's duration is attributable to the agent span that awaited it. When it is violated, three operational consequences follow: (a) trace-based latency analysis undercounts agent time because the async leg lands in a separate root; (b) sampling becomes inconsistent across the boundary, so partial traces are the norm rather than the exception; (c) incident reconstruction fails precisely at the async boundary — the place where most production failures occur — because that is the subtree that gets orphaned. The failure is invisible at write time and only surfaces under trace-backend queries, which is why it is routinely misdiagnosed as a backend problem rather than a propagation-ownership problem. ```python from dataclasses import dataclass from typing import Optional, Mapping class TraceContextOrphanError(RuntimeError): """Raised when a child span would be parented to a span that is no longer live.""" pass @dataclass(frozen=True) class SpanHandle: trace_id: str span_id: str sampled: bool trace_state: str is_open: bool def carrier_from_span(span: SpanHandle) -> Mapping[str, str]: # Full carrier: dropping trace_flags or trace_state is the root-split bug. return { "traceparent": f"00-{span.trace_id}-{span.span_id}-{'01' if span.sampled else '00'}", "tracestate": span.trace_state, } def create_child_span( parent: SpanHandle, name: str, *, allow_link_fallback: bool, ) -> SpanHandle: """Create a child span across an async tool-call boundary. The parent agent span may be suspended; it must still be OPEN to be a valid parent. If it is closed, we refuse to fabricate a parent and require link semantics instead. """ if not parent.is_open: if not allow_link_fallback: raise TraceContextOrphanError( f"cannot parent span {name!r} to closed span {parent.span_id}; " "keep the agent span open across the await, or use link semantics" ) # Link semantics: no parent_id, record a link to the closed span. return SpanHandle( trace_id=parent.trace_id, span_id=_new_span_id(), sampled=parent.sampled, trace_state=parent.trace_state, is_open=True, ) if not parent.sampled and parent.trace_state == "": # Guard against silently dropping the sampled bit during handoff. raise TraceContextOrphanError( f"carrier for span {parent.span_id} lost trace_flags/trace_state; " "worker would start a new root" ) return SpanHandle( trace_id=parent.trace_id, span_id=_new_span_id(), sampled=parent.sampled, trace_state=parent.trace_state, is_open=True, ) def _new_span_id() -> str: import secrets return secrets.token_hex(8) def suspend_agent_span(span: SpanHandle) -> SpanHandle: """Mark the agent span suspended but still OPEN — do not end it here.""" if not span.is_open: raise TraceContextOrphanError("suspend on an already-closed span") return SpanHandle( trace_id=span.trace_id, span_id=span.span_id, sampled=span.sampled, trace_state=span.trace_state, is_open=True, # stays open across the async boundary ) ```