# What happens when an instance of an agent delegation chain loses its parent span link: orphaned GenAI semantic convention spans at the second-level delegation boundary **Date of Record:** 2026 November 18 **Scope:** Applies to OpenTelemetry-instrumented multi-agent systems where GenAI semantic conventions (`gen_ai.*` attributes) are emitted across two or more levels of agent delegation, and where each delegation creates a new span whose parent is expected to be the delegating agent's span. Does not apply to flat single-agent tool-call traces, nor to systems that propagate a W3C `traceparent` header through every delegation hop as a hard requirement. ## Core Distinction There is a difference between *trace context propagation* (the W3C `traceparent`/`tracestate` mechanism that carries trace ID, span ID, and flags across process boundaries) and *semantic span linking* (the `gen_ai.*` attributes and explicit `SpanLink` objects that record *why* a span exists and *which* prior span caused it). A system can have perfect context propagation — every span shares the same trace ID — while simultaneously having broken semantic linkage, because the parent-child relationship is inferred from the active context at span creation time, and that context can be silently replaced by an intermediate agent's own span lifecycle. The non-obvious boundary: at the **second delegation hop**, an intermediate agent that runs its own span as a *root* (rather than as a child of the inbound request span) produces a structurally valid trace with a single trace ID but a **detached subtree**. The leaf agent's span then has a parent that is *not* the span that semantically delegated to it. The trace looks complete. The causal graph does not. ## What Was Observed In a three-tier delegation (Orchestrator → Planner → Executor), with each tier instrumented independently: 1. Orchestrator creates span `A` (root), sets `gen_ai.request.model`, and calls Planner with a propagated `traceparent` pointing to `A`. 2. Planner's instrumentation starts span `B` using the inbound context as parent — correct, `B.parent = A`. 3. Planner performs an internal *re-planning loop* that calls `tracer.start_as_current_span(...)` without `context=` argument after an `asyncio.gather` boundary. The `contextvars` copy is dropped at the task boundary. Span `B` remains open, but the *current* context during the Executor call is empty. 4. Executor's instrumentation, seeing an empty context, starts span `C` as a **new root** with a fresh trace ID — or, in the variant where the SDK is configured with `OTEL_PROPAGATORS=none` fallback, as a sibling root under the *same* trace ID but with `parent_span_id` unset. 5. The delegation from Planner to Executor is recorded only as a `gen_ai.agent.delegation` *event* on `B`, not as a parent relationship on `C`. Result: the causal chain `A → B → C` is present in the event log but absent in the span tree. Any query that walks `parent_span_id` — latency attribution, cost rollup by delegation depth, error blast-radius — silently undercounts. The Executor's tokens are billed to a phantom root. ## Condition of Validity This failure is *structurally latent* but only *materializes* under a conjunction of conditions: - The intermediate agent crosses an async task boundary (`asyncio.create_task`, thread pool handoff, or Celery/queue hop) between receiving the inbound context and starting the outbound delegation span. - The instrumentation library does **not** re-inject `SpanLink` objects referencing the delegating span when the parent-child relationship cannot be established. - The GenAI semantic conventions in use predate the `gen_ai.agent.id` / `gen_ai.conversation.id` linkage attributes, or the implementation does not emit them. If any one of these is false, the linkage survives. If all three are true, the trace is *numerically complete and semantically broken* — the worst category, because dashboards built on span counts and trace-ID grouping will show green. ## Decision Boundary The operational decision is whether to treat an orphaned delegation span as an **error** or as a **degraded-but-acceptable** record. - **Treat as error** when span linkage is load-bearing for billing, SLO attribution, or safety auditing (e.g., attributing a harmful tool call to the originating user request). In these contexts an orphaned span is a *correctness* defect, not an observability defect, and must fail the request or at minimum fail the trace-validation gate. - **Treat as degraded** when linkage is used only for exploratory latency analysis, where a `SpanLink` fallback with `gen_ai.agent.delegation.source_span_id` is sufficient to reconstruct causality offline. The boundary is crossed the moment any downstream consumer *assumes* tree structure to compute an aggregate. That assumption is the invariant: **every `gen_ai` span at delegation depth ≥ 2 must either have a resolvable parent within the same trace or carry an explicit link to its delegating span.** If neither holds, the span is orphaned and the aggregate is untrustworthy. ## Operational Effect Concretely, in the observed incident: - Cost rollup by `gen_ai.usage.input_tokens` grouped by root span undercounted Planner+Executor spend by ~31% because Executor spans were attributed to a separate root. - p99 latency for the orchestrator was reported as 2.1s; the true end-to-end including Executor was 7.4s. The gap was invisible because the Executor subtree was a disconnected trace. - A safety review could not attribute a flagged tool invocation to its originating user session, because the session ID attribute lived on span `A` and the flagged span `C` shared no resolvable path back to it. The fix that held: at every delegation boundary, the delegating code must explicitly pass `context=` derived from the *inbound* span (not the currently-active span, which may be the intermediate agent's own), and instrumentation must emit a `SpanLink` to the delegating span whenever `parent_span_id` cannot be set. Validation gates reject traces where a span has `gen_ai.agent.delegation.depth > 1` and neither a parent nor a link. The following validator encodes the invariant as an executable check and raises on violation, so the failure surfaces at trace-emission time rather than at rollup time. ```python from __future__ import annotations from dataclasses import dataclass, field from typing import Optional class OrphanedDelegationSpanError(RuntimeError): """Raised when a gen_ai delegation span has neither a resolvable parent within its trace nor an explicit SpanLink to its delegating span.""" @dataclass class SpanRecord: span_id: str trace_id: str parent_span_id: Optional[str] delegation_depth: int delegating_span_id: Optional[str] = None # from explicit SpanLink attributes: dict = field(default_factory=dict) def validate_delegation_linkage(spans: list[SpanRecord]) -> None: """Enforce: every span at delegation depth >= 2 must have either a parent in the same trace or an explicit link to its delegating span.""" by_id = {s.span_id: s for s in spans} for span in spans: if span.delegation_depth < 2: continue parent_resolves = ( span.parent_span_id is not None and span.parent_span_id in by_id and by_id[span.parent_span_id].trace_id == span.trace_id ) link_resolves = ( span.delegating_span_id is not None and span.delegating_span_id in by_id and by_id[span.delegating_span_id].trace_id == span.trace_id ) if not (parent_resolves or link_resolves): raise OrphanedDelegationSpanError( f"span {span.span_id} (trace {span.trace_id}, depth " f"{span.delegation_depth}) has no resolvable parent " f"(parent_span_id={span.parent_span_id!r}) and no resolvable " f"delegating SpanLink (delegating_span_id=" f"{span.delegating_span_id!r}); causal graph is broken" ) def emit_delegation_span( *, span_id: str, trace_id: str, parent_span_id: Optional[str], delegation_depth: int, delegating_span_id: Optional[str], ) -> SpanRecord: record = SpanRecord( span_id=span_id, trace_id=trace_id, parent_span_id=parent_span_id, delegation_depth=delegation_depth, delegating_span_id=delegating_span_id, ) # Validate this span against the invariant before it leaves the process. validate_delegation_linkage([record]) return record ```