# Why does an instance of OpenTelemetry GenAI token accounting silently double-count tool-call output when `gen_ai.usage.output_tokens` is emitted on both the parent agent span and the child tool span **Date of Record:** 2026 November 18 **Scope:** Applies to OTel GenAI semantic conventions v1.28+ where agent spans (`gen_ai.operation.name = "invoke_agent"`) and tool spans (`gen_ai.operation.name = "execute_tool"`) coexist under a single trace, and where a downstream collector or billing pipeline aggregates `gen_ai.usage.output_tokens` by summing across spans. Does not apply to single-span LLM call instrumentation (`chat`) or to pipelines that aggregate strictly by leaf span. ## Core Distinction The GenAI semantic conventions define `gen_ai.usage.output_tokens` as an attribute of the span that *represents the model inference*. In a chat-only instrumentation, there is exactly one such span per call, and summation is trivially correct. In an agentic instrumentation, the same attribute is legitimately emitted on the `invoke_agent` span (the aggregate of the whole agent turn, including all model calls it triggered) *and* on each `execute_tool` span when the tool itself performs an internal model call (e.g., a retrieval tool that runs a reranker LLM, or a code-interpreter tool that invokes a model to summarize). The convention does not forbid this; it only says the attribute should reflect the output tokens attributable to that span's operation. The boundary failure is that "attributable to this span" is inclusive at the agent level and exclusive at the tool level, and no attribute in the spec disambiguates the two. A naive sum is therefore not a sum of disjoint quantities — it is a sum of a nested aggregate and its constituents. ## What Was Observed On a production trace of a research agent (single `invoke_agent` span, three `execute_tool` children — `web_search`, `fetch_page`, `summarize`), the collector's token rollup reported 14,820 output tokens for a turn whose true billed output was 9,240. The delta was exactly the sum of `gen_ai.usage.output_tokens` on the two tool spans that internally invoked a model. The parent `invoke_agent` span's attribute already included those tool-internal generations because the agent framework computed the turn total after all tool calls returned. Both spans carried `gen_ai.usage.output_tokens`, both were tagged with the same `gen_ai.response.model`, and neither carried any marker distinguishing "aggregate" from "leaf." The double-count was stable and reproducible across 100% of turns containing a model-backed tool, and absent on turns with only non-model tools — which is why it survived canary testing: the canary workload used a `calculator` tool. ## Condition of Validity The invariant "sum of `gen_ai.usage.output_tokens` over all spans in a trace equals billed output tokens" holds **only if** no span is an ancestor of another span that also carries the attribute. Equivalently: the attribute must form an antichain over the span tree. This is satisfied when (a) only `chat` spans carry the attribute, or (b) `invoke_agent` spans carry a *separate* aggregate attribute (e.g., a custom `gen_ai.usage.agent.total_output_tokens`) and leave `gen_ai.usage.output_tokens` unset. It is violated whenever an agent span and any descendant both emit `gen_ai.usage.output_tokens`. The convention's silence on nesting makes the violating configuration spec-legal, so validation against the spec will not catch it. ## Decision Boundary If you control instrumentation: emit `gen_ai.usage.output_tokens` only on leaf spans (spans with no descendant that also emits it), and expose the agent-turn total under a distinct, non-convention attribute. If you consume third-party instrumentation: do not sum flat. Aggregate by walking the span tree and, at each node, take `max(own_attribute, sum(children_attributes))` rather than `own_attribute + sum(children_attributes)` — this assumes the parent's value is the inclusive aggregate, which is the observed behavior of the major agent SDKs. If neither is possible, filter the rollup to spans where `gen_ai.operation.name == "chat"` and accept that model-backed tools instrumented as `execute_tool` will be undercounted rather than double-counted. The choice between over- and under-counting is a billing-integrity decision, not a technical one; document it. ## Operational Effect Undetected, the failure inflates cost dashboards proportionally to tool-model density, which biases optimization toward the wrong tools and can trigger false budget alarms during tool-heavy turns. It also corrupts any per-tool cost attribution that divides parent tokens among children. Detection is cheap: compute, per trace, `sum(all spans) - max_over_antichain` and alert when the residual is nonzero and correlates with the count of model-backed tool spans. The residual is a direct measure of nesting depth of the attribute, and trending it catches instrumentation regressions the moment a tool is upgraded to include an internal model call. ```python class TokenAccountingInvariantError(RuntimeError): """Raised when gen_ai.usage.output_tokens does not form an antichain over the span tree.""" def rollup_output_tokens(spans: list[dict]) -> int: """ spans: list of dicts with keys 'span_id', 'parent_span_id', 'operation', and optional 'output_tokens'. Returns the deduplicated output-token total, treating a parent's value as an inclusive aggregate when a descendant also carries the attribute. """ by_id = {s["span_id"]: s for s in spans} children: dict[str | None, list[str]] = {} for s in spans: children.setdefault(s.get("parent_span_id"), []).append(s["span_id"]) def subtree_has_token(span_id: str) -> bool: s = by_id[span_id] if s.get("output_tokens") is not None: return True return any(subtree_has_token(c) for c in children.get(span_id, [])) def own(span_id: str) -> int: return by_id[span_id].get("output_tokens") or 0 def resolve(span_id: str) -> int: kids = children.get(span_id, []) child_sum = sum(resolve(c) for c in kids) if own(span_id) and child_sum: # Parent carries an inclusive aggregate; do not add. return max(own(span_id), child_sum) return own(span_id) + child_sum roots = children.get(None, []) if not roots: raise TokenAccountingInvariantError( "no root span found; trace is malformed and token rollup is undefined" ) # Validate the antichain assumption before trusting the max() collapse. for s in spans: if own(s["span_id"]) and any( subtree_has_token(c) for c in children.get(s["span_id"], []) ): # Legal but ambiguous: parent aggregate overlaps a child leaf. # We resolve via max(); flag if the parent is *smaller* than # the child sum, which means it is not actually inclusive. child_sum = sum(resolve(c) for c in children.get(s["span_id"], [])) if own(s["span_id"]) < child_sum: raise TokenAccountingInvariantError( f"span {s['span_id']} ({s.get('operation')}) carries " f"output_tokens={own(s['span_id'])} but its descendants " f"sum to {child_sum}; the parent is not an inclusive " f"aggregate and flat summation would be required" ) return sum(resolve(r) for r in roots) ```