# Why does an instance of the OpenTelemetry GenAI agent span silently lose tool-call attributes and token/cost metrics when a sub-agent executes a tool call **Date of Record:** 2026 November 16 **Scope:** This record applies to services instrumenting LLM agent workflows with the OpenTelemetry GenAI semantic conventions (spans of kind `agent`, `tool`, and `chat`/`inference`, plus the `gen_ai.usage.*` metrics that ride on them), specifically when tool execution is delegated to a nested agent or sub-agent process. It does not apply to single-process, single-agent traces where the tool call is emitted by the same span that owns the `gen_ai.usage.input_tokens` / `output_tokens` attributes, nor to pure HTTP/gRPC propagation without agent nesting. ## Core Distinction There are two structurally different things that both look like "a tool call happened": 1. **An agent span that *declares* a tool call** — the parent `agent` span carries `gen_ai.operation.name = "execute_tool"`, `gen_ai.tool.name`, `gen_ai.tool.call.id`, and the associated `gen_ai.usage.*` token attributes. This is the span the cost pipeline reads. 2. **A tool span that *performs* the call** — a child span (often of kind `tool`, often emitted by a different process or SDK instance) that actually runs the function and returns a result. The GenAI conventions assume these collapse into one record. The failure boundary is that they do **not** collapse when the tool executes inside a sub-agent that was started with a *new* trace context (or a fresh `TracerProvider`) rather than a propagated one. The parent `agent` span then owns the tool *declaration* and the token/cost attributes; the child `tool` span owns the *execution* and the result. If anything in the pipeline keys cost attribution on the `tool` span — or keys tool-call reconstruction on the `agent` span — the two halves are orphaned from each other. The non-obvious invariant: **`gen_ai.usage.*` is a property of the span that consumed tokens, and tool-call attributes are a property of the span that invoked the tool; when those are different spans, the conventions do not define which one is authoritative, and every downstream consumer picks a different answer.** ## What Was Observed In a multi-agent deployment (orchestrator agent → planner sub-agent → tool executor), traces showed the following pattern after the sub-agent was refactored to run behind a process boundary: - The orchestrator `agent` span correctly carried `gen_ai.operation.name`, `gen_ai.agent.name`, and `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` for the planning turn. - The `tool` span (kind `tool`) appeared as a child of the *sub-agent's* span, not the orchestrator's, because the sub-agent created a fresh `TracerProvider` and did not extract the incoming `traceparent`. - `gen_ai.tool.call.id` was present on the tool span, but the matching `gen_ai.tool.call.id` on the agent span was absent — the agent span only had `gen_ai.tool.name`. - Cost dashboards that joined `gen_ai.usage.*` to `gen_ai.tool.name` **double-counted**: the orchestrator span attributed tokens to the tool name, and the sub-agent span attributed *its* tokens to the same tool name, producing a 2× cost inflation for every delegated call. - Latency dashboards that joined on `gen_ai.tool.call.id` **under-counted to zero**: no single span carried both the call id and the token usage, so the join produced empty rows and the tool appeared to cost nothing. Critically, nothing errored. The spans were valid, the attributes were spec-conformant, and the trace was structurally legal. The loss was purely in the *relational* layer between spans. A second, subtler observation: when the sub-agent ran in-process but with `BatchSpanProcessor` flushed at different times than the parent, the child `tool` span occasionally arrived at the collector *before* its parent `agent` span. Backends that assume parent-before-child ordering for attribute resolution (some columnar trace stores) resolved `gen_ai.tool.call.id` to null on the parent during the first pass and never reconciled it on the second. ## Condition of Validity The failure is guaranteed to manifest when **all** of the following hold: 1. Tool execution is delegated across a context boundary (process, thread with a fresh context, worker pool, remote sub-agent) and the `traceparent` / `baggage` is not explicitly extracted and re-injected. 2. The parent `agent` span sets `gen_ai.usage.*` (because the orchestrator's LLM turn produced the tool-call decision and consumed tokens) **and** the child `tool` span also sets or inherits usage attributes. 3. A downstream consumer performs a join on either `gen_ai.tool.call.id` (expecting it on the agent span) or on `gen_ai.tool.name` (expecting uniqueness per trace). It does **not** manifest when: the sub-agent shares the parent's `TracerProvider` and context (single-process, context-propagated); when usage attributes are set only on the `chat`/`inference` span and never duplicated on `agent`/`tool` spans; or when the backend keys everything on `trace_id` alone and tolerates attribute asymmetry. ## Decision Boundary When you cannot guarantee context propagation across the tool-execution boundary, you must choose one authoritative span for cost and one for tool identity, and enforce it: - **Cost-authoritative span:** the `chat`/`inference` span (the span that literally called the model). Never set `gen_ai.usage.*` on an `agent` span that also emits a child `tool` span; instead, let the inference span carry usage and join tool identity via `gen_ai.tool.call.id` set on *both* the inference span and the tool span. - **Tool-identity span:** the `tool` span, with `gen_ai.tool.call.id` and `gen_ai.tool.name` set, and *no* usage attributes. If you cannot restructure the instrumentation, the fallback boundary is: **propagate `traceparent` explicitly** at the sub-agent boundary and accept that the parent agent span will show usage *including* the sub-agent's tokens (double-counted relative to the sub-agent's own inference span). In that case, deduplicate at query time on `(trace_id, gen_ai.tool.call.id)` and take the *minimum* usage across spans sharing that call id — never the sum. The invariant to encode in your pipeline: **for any `gen_ai.tool.call.id`, exactly one span in the trace may carry non-zero `gen_ai.usage.*`, and it must be the span whose `gen_ai.operation.name` is `"chat"` (or `"inference"`), not `"execute_tool"`.** Violation of this invariant is the signal that delegation broke context propagation. ## Operational Effect The practical consequence is a *silent* cost-attribution corruption that survives schema validation, passes span-level linting, and only surfaces as a statistical anomaly (2× inflation on one dashboard, zero on another) weeks later. Because the spans are individually valid, no alert fires on span emission. The corruption is only detectable by a cross-span relational check, which must be run as a post-collection audit rather than a per-span validator. The remediation is cheap (re-inject `traceparent`, or move usage off the agent span) but is almost never applied because the symptom is invisible at the point of instrumentation. The operational lesson: treat the GenAI agent/tool span split as a *relational contract*, not a per-span schema, and enforce the single-usage-span invariant with an explicit audit that raises on violation. ```python from __future__ import annotations from dataclasses import dataclass class GenAIUsageAttributionError(RuntimeError): """Raised when a trace violates the single-usage-span invariant for a tool call.""" @dataclass(frozen=True) class SpanRecord: trace_id: str span_id: str operation_name: str | None # gen_ai.operation.name tool_call_id: str | None # gen_ai.tool.call.id input_tokens: int # gen_ai.usage.input_tokens output_tokens: int # gen_ai.usage.output_tokens _USAGE_OWNER_OPS = frozenset({"chat", "inference"}) def enforce_single_usage_span(spans: list[SpanRecord]) -> None: """ For every (trace_id, tool_call_id), assert exactly one span carries usage and that span's operation is a chat/inference span, never execute_tool. """ by_call: dict[tuple[str, str], list[SpanRecord]] = {} for s in spans: if s.tool_call_id is None: continue if s.input_tokens or s.output_tokens: by_call.setdefault((s.trace_id, s.tool_call_id), []).append(s) for (trace_id, call_id), owners in by_call.items(): if len(owners) > 1: raise GenAIUsageAttributionError( f"trace {trace_id} call {call_id}: {len(owners)} spans carry " f"gen_ai.usage.* ({[o.span_id for o in owners]}); " "context propagation across the tool boundary is likely broken " "and cost will be double-counted." ) owner = owners[0] if owner.operation_name not in _USAGE_OWNER_OPS: raise GenAIUsageAttributionError( f"trace {trace_id} call {call_id}: usage carried by span " f"{owner.span_id} with operation {owner.operation_name!r}; " "usage must live on a chat/inference span, not on the " "agent/tool span that merely declared the call." ) ```