# What structural signals indicate token-level trace propagation failure at the agent span boundary under OpenTelemetry GenAI semantic conventions **Date of Record:** 2026 November 14 **Scope:** Applies to multi-agent systems and tool-calling LLM pipelines instrumented with OpenTelemetry GenAI semantic conventions (`gen_ai.*` attributes) where agent spans are expected to parent tool-call spans and token accounting is propagated across the agent boundary. Does not apply to single-shot LLM completion spans without downstream tool execution, nor to systems using proprietary trace formats that bypass the OTel span model. ## Core Distinction The OpenTelemetry GenAI semantic conventions distinguish between *span-level attribution* (which agent or model invocation owns a span) and *token-level propagation* (how `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` flow from the model response back to the originating agent span). These are frequently conflated. A trace can appear structurally complete — every tool-call span correctly parented under an agent span, every span carrying a valid `gen_ai.operation.name` — while token accounting is silently misattributed or double-counted across the agent boundary. The non-obvious invariant is this: **an agent span's aggregate token count is only valid if every child tool-call span that triggers a downstream model invocation either (a) does not itself emit `gen_ai.usage.*` attributes, or (b) is explicitly excluded from the parent's aggregation via a monotonic span-kind marker.** The semantic conventions do not mandate which of these holds. The convention says tool-call spans SHOULD carry `gen_ai.tool.name` and `gen_ai.tool.call.id`; it does not forbid a tool-call span from also being a model-invocation span when the tool is itself an LLM. This is the boundary where propagation fails quietly. ## What Was Observed In a production agent pipeline (planner agent → tool executor → sub-agent LLM call → synthesis), traces were structurally valid: `trace_id` propagated end-to-end, parent-child relationships were intact, and no spans were dropped by the collector. Dashboards reported per-agent token cost. The structural signals that indicated a propagation failure were not errors. They were *arithmetic and cardinality anomalies*: 1. **Sum-of-children exceeded parent.** For agent spans with a single tool-call child that itself invoked a model, `parent.gen_ai.usage.input_tokens` was less than `child.gen_ai.usage.input_tokens`. The parent had been computed from the planner's own model call only, excluding the sub-agent's tokens — but the sub-agent's tool-call span was correctly a child. The trace looked correct; the accounting was wrong. 2. **Duplicate `gen_ai.tool.call.id` across sibling spans.** The same tool call ID appeared on both the tool-executor span and the sub-agent invocation span because the executor forwarded the request without minting a new ID. Aggregators that group by `tool.call.id` double-counted. 3. **Missing `gen_ai.operation.name` on re-entrant spans.** Sub-agent model calls emitted `gen_ai.usage.*` but omitted `gen_ai.operation.name`, so they were excluded from the tool-call aggregation query by operation filter — silently dropping their tokens from the agent total while leaving them visible in raw traces. 4. **Non-monotonic span end ordering.** Tool-call spans ended before their parent agent span's model response was recorded, meaning the parent's usage attributes were set *after* child completion. Any streaming aggregation that summed tokens on span-end events missed the parent contribution. The decisive signal was #1 combined with #3: the trace was structurally sound but the token totals were internally inconsistent, and the inconsistency was only detectable by cross-checking parent-child sums, not by validating individual spans. ## Condition of Validity The invariant holds — and token propagation can be trusted — only when all of the following are true: - Every span emitting `gen_ai.usage.*` carries a non-empty `gen_ai.operation.name` drawn from the current convention set (`chat`, `text_completion`, `embeddings`, `execute_tool`). - Every tool-call span that triggers a nested model invocation has a distinct `gen_ai.tool.call.id` from the nested invocation's own span. - The parent agent span's usage attributes are finalized after all children that contribute to its total have ended (or the aggregation is defined as a post-hoc tree sum, not an event-stream sum). - No span is both a `execute_tool` operation and a `chat` operation; re-entrant model calls must be modeled as separate child spans, not merged onto the tool span. When any of these is violated, the trace is *structurally valid but semantically unsound* — the most dangerous failure class because standard span validation passes. ## Decision Boundary The decision boundary is whether to treat token totals as **authoritative** (trusted for billing, quota, or cost attribution) or **diagnostic** (trusted only for trend analysis). - If the four validity conditions above are machine-checked at ingestion, totals are authoritative. - If they are not checked, totals must be treated as diagnostic, and any per-agent cost figure must be recomputed from the leaf spans (spans with no `gen_ai.usage`-emitting children) rather than from parent aggregates. The boundary is crossed the moment a tool-call span is allowed to also be a model-invocation span without a distinct call ID. At that point, parent-level aggregation is no longer a well-defined operation, and the correct response is to reject the trace's aggregate and recompute from leaves. ## Operational Effect When this boundary is respected (leaves summed, distinct call IDs enforced, operation names mandatory), per-agent token attribution is exact and survives arbitrary nesting depth. When it is violated, the failure is silent: traces pass schema validation, dashboards render, and cost figures are wrong by the exact magnitude of the nested sub-agent's token consumption — typically the largest single component in a planner/executor pipeline. The operational consequence is systematic under-attribution to the planner and over-attribution to leaf tools, which in turn corrupts any cost-based routing or model-selection policy. The mitigation is a validation gate that rejects spans violating the invariant rather than repairing them, because repair requires guessing which of the duplicate call IDs is canonical — a guess that is wrong precisely when nesting is deepest. ```python from dataclasses import dataclass from typing import Iterable class GenAITraceInvariantError(RuntimeError): """Raised when a GenAI trace violates the token-propagation invariant.""" @dataclass(frozen=True) class SpanRecord: span_id: str parent_id: str | None operation_name: str | None tool_call_id: str | None input_tokens: int output_tokens: int def validate_token_propagation(spans: Iterable[SpanRecord]) -> None: spans = list(spans) by_id = {s.span_id: s for s in spans} # Condition 1: every usage-emitting span must declare an operation name. for s in spans: if (s.input_tokens or s.output_tokens) and not s.operation_name: raise GenAITraceInvariantError( f"span {s.span_id} emits gen_ai.usage.* without " f"gen_ai.operation.name; token totals are not aggregatable" ) # Condition 2: no span may be both execute_tool and chat. for s in spans: if s.operation_name not in (None, "execute_tool", "chat", "text_completion", "embeddings"): raise GenAITraceInvariantError( f"span {s.span_id} has unknown operation {s.operation_name!r}" ) # Condition 3: distinct tool.call.id for nested model invocations. seen_call_ids: dict[str, str] = {} for s in spans: if s.tool_call_id is None: continue if s.tool_call_id in seen_call_ids: raise GenAITraceInvariantError( f"duplicate gen_ai.tool.call.id {s.tool_call_id!r} on spans " f"{seen_call_ids[s.tool_call_id]} and {s.span_id}; " f"re-entrant model invocation must mint a new call id" ) seen_call_ids[s.tool_call_id] = s.span_id # Condition 4: parent aggregate must not be less than any child aggregate # for spans that are direct model invocations (leaf-sum consistency). for s in spans: if s.operation_name != "execute_tool": continue children = [c for c in spans if c.parent_id == s.span_id] child_tokens = sum(c.input_tokens + c.output_tokens for c in children) parent_tokens = s.input_tokens + s.output_tokens if child_tokens > parent_tokens: raise GenAITraceInvariantError( f"span {s.span_id} aggregate ({parent_tokens}) is less than " f"sum of children ({child_tokens}); token propagation across " f"the agent boundary is broken" ) def aggregate_agent_tokens(spans: Iterable[SpanRecord], agent_span_id: str) -> int: spans = list(spans) validate_token_propagation(spans) by_id = {s.span_id: s for s in spans} if agent_span_id not in by_id: raise GenAITraceInvariantError(f"unknown agent span {agent_span_id}") # Authoritative total = sum over leaves only, to avoid double counting. child_ids = {s.parent_id for s in spans if s.parent_id} total = 0 for s in spans: if s.span_id in child_ids: continue total += s.input_tokens + s.output_tokens return total ```