# Why does an instance of an agent span fail the GenAI semantic convention when tool-call telemetry crosses the parent/child boundary mid-turn **Date of Record:** 2026 November 16 **Scope:** Applies to OpenTelemetry instrumentation of LLM agent runtimes that emit `gen_ai.*` spans for a single logical turn where one or more tool/function calls occur, and where the tool execution is represented as a child span of the agent span. Does not apply to stateless single-shot completion calls, nor to agent frameworks that flatten tool calls into attributes on the parent span without emitting child spans. ## Core Distinction There is a difference between *an agent span that contains tool calls as attributes* and *an agent span whose tool calls are emitted as child spans*. The GenAI semantic conventions define attributes like `gen_ai.operation.name`, `gen_ai.agent.name`, `gen_ai.tool.name`, `gen_ai.tool.call.id`, and `gen_ai.usage.input_tokens`/`output_tokens`. The convention assumes a *linear* accounting model: one model invocation produces one usage record, and tool calls are discrete events with a request and a response. The failure mode is this: when an agent turn performs N model calls interleaved with M tool calls, the parent agent span cannot carry a single coherent `gen_ai.usage.*` value, because usage accrues per model invocation, not per turn. Instrumentations that "helpfully" sum token usage onto the parent agent span produce a span whose `gen_ai.usage.input_tokens` is *not* the usage of any single operation. Downstream cost aggregators, budget enforcers, and rate-limit attribution systems that read that attribute will double-count or mis-attribute, because the child model spans *also* carry the same usage. The invariant that is violated is: **for any span carrying `gen_ai.operation.name`, `gen_ai.usage.*` must describe exactly the model invocation that span represents — never an aggregate over descendants.** ## What Was Observed In a production agent runtime (ReAct-style loop, max 8 tool iterations), the following was observed after enabling an OTel GenAI exporter: - The root agent span (`gen_ai.operation.name=invoke_agent`) carried `gen_ai.usage.input_tokens=14820`, `gen_ai.usage.output_tokens=2310`. - Each of the 6 child `gen_ai.operation.name=chat` spans carried its own usage (e.g., 2410/380, 2501/402, ...), summing to exactly the parent's value. - A cost dashboard that summed `gen_ai.usage.input_tokens` across *all* spans in a trace reported roughly **2×** the true token spend, because it counted both parent aggregate and children. - A separate budget-enforcement service that keyed on the agent span only reported the *correct* number but could not attribute it to a model or a provider, because the parent span had no single `gen_ai.request.model` (six different model IDs appeared across the turn due to a fallback chain). The non-obvious part: the double-count was **not** flagged by any schema validation. The parent span is *valid* under the semconv — `gen_ai.usage.*` is an allowed attribute on agent spans. The convention permits it. The failure is semantic, not structural, and therefore invisible to linting. A second, subtler observation: `gen_ai.tool.call.id` was generated by the agent framework locally and did **not** match the `tool_call_id` returned by the model in the assistant message. When the tool span was correlated back to the model's tool-call request, the join failed silently, and the tool latency was attributed to the wrong call in a retry scenario (the model re-issued the same tool call with a new id after a transient 503). ## Condition of Validity The invariant "usage attributes are per-operation, not per-subtree" holds only when: 1. Every model invocation emits exactly one span with `gen_ai.operation.name` in `{chat, text_completion, generate_content}` and that span carries its own `gen_ai.usage.*`. 2. The agent span does **not** also carry `gen_ai.usage.*` unless the turn contained exactly one model invocation (in which case parent and child are identical and the parent copy is redundant but harmless). 3. `gen_ai.tool.call.id` on the tool span is copied verbatim from the model's tool-call response, not regenerated by the framework. 4. Tool spans are children of the *model span that requested them*, not children of the agent turn span, so that causality (model → tool → model) is preserved in the trace tree. If condition 2 is violated, aggregation consumers double-count. If condition 4 is violated, the trace shows tool calls as siblings of the model calls that issued them, and critical-path analysis of "which model call was slow because of a tool" becomes impossible. ## Decision Boundary Choose between two representations at instrumentation time, and do not mix them within one agent runtime: - **Aggregate-on-parent representation:** the agent span carries summed usage and a `gen_ai.agent.name`; child model spans carry per-call usage but are marked `gen_ai.operation.name=chat` with no agent attributes. Consumers must be told to read usage from *either* the agent span *or* the leaf spans, never both. This is acceptable only if you control every consumer. - **Leaf-only representation (preferred):** the agent span carries *no* `gen_ai.usage.*`; usage lives exclusively on model-invocation leaf spans. Aggregators sum leaves. This is the only representation that is safe against arbitrary third-party consumers, because it does not create two valid sources of truth. The boundary is crossed the moment a second team's dashboard reads your spans without knowing your convention. At that point the aggregate-on-parent representation becomes a latent double-count bug. ## Operational Effect After migrating to leaf-only usage and reparenting tool spans under their issuing model spans: - Cost dashboards matched ground-truth billing within 0.3% (residual from provider-side rounding). - Budget enforcement could attribute spend per model and per provider fallback, enabling a per-model circuit breaker. - Retry-induced duplicate tool calls became visible as distinct `gen_ai.tool.call.id` values under the same model span, exposing a retry storm that had previously been invisible. The cost was a short window of dashboard under-reporting during rollout, because the leaf-only change removed the parent aggregate before consumers were updated. This is the correct failure direction: under-reporting is detectable against billing; double-counting is not. ```python class GenAIUsageInvariantError(RuntimeError): """Raised when a span's gen_ai.usage.* attributes violate the per-operation invariant.""" def __init__(self, span_name: str, reason: str) -> None: super().__init__(f"span={span_name!r}: {reason}") self.span_name = span_name self.reason = reason def validate_agent_span(span: dict) -> None: """ Enforce: a span carrying gen_ai.operation.name == 'invoke_agent' must NOT carry gen_ai.usage.* unless the turn had exactly one model invocation. """ attrs = span.get("attributes", {}) op = attrs.get("gen_ai.operation.name") carries_usage = any( k.startswith("gen_ai.usage.") for k in attrs ) child_model_spans = [ c for c in span.get("children", []) if c.get("attributes", {}).get("gen_ai.operation.name") in {"chat", "text_completion", "generate_content"} ] if op == "invoke_agent" and carries_usage and len(child_model_spans) > 1: raise GenAIUsageInvariantError( span.get("name", ""), f"agent span carries aggregate gen_ai.usage.* across " f"{len(child_model_spans)} child model spans; move usage to leaf spans", ) if op == "invoke_agent" and carries_usage and len(child_model_spans) == 1: # Redundant but harmless; warn by returning without raising. return for child in span.get("children", []): validate_agent_span(child) def validate_tool_call_id(model_span: dict, tool_span: dict) -> None: """ Enforce: tool span's gen_ai.tool.call.id must equal the id the model issued. """ issued = { tc["id"] for tc in model_span.get("attributes", {}).get("gen_ai.response.tool_calls", []) } used = tool_span.get("attributes", {}).get("gen_ai.tool.call.id") if used not in issued: raise GenAIUsageInvariantError( tool_span.get("name", ""), f"gen_ai.tool.call.id={used!r} not present in issuing model span ids={sorted(issued)}", ) ```