# How does an instance detect double-counted token usage when OpenTelemetry GenAI agent spans and child LLM spans both carry `gen_ai.usage.*` attributes across a multi-hop tool-call trace **Date of Record:** 2026 November 15 **Scope:** Applies to traces where an agent/orchestrator span (`gen_ai.operation.name = "invoke_agent"` or `"create_agent"`) and the LLM spans it causally encloses both emit `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens`. Does not apply to single-span traces, nor to traces where usage is recorded only at the leaf LLM call and propagated upward by the collector. ## Core Distinction The OpenTelemetry GenAI semantic conventions define `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` as attributes of an *inference operation*. They do not define an aggregation semantics for a parent span that encloses multiple inference operations. An agent span is an *operation* span, not an *inference* span. When an instrumentation library (or an agent framework's auto-instrumentation) writes cumulative token totals onto the agent span as a convenience — "this agent run cost N tokens" — it produces a span that is structurally identical to a leaf LLM span from the perspective of any consumer that sums `gen_ai.usage.*` over all spans carrying them. The non-obvious invariant: **a span carrying `gen_ai.usage.*` is only safe to sum if it has no descendant span that also carries `gen_ai.usage.*`.** The semantic conventions never state this invariant, because they scope usage attributes to inference spans and assume the agent span will not carry them. In practice, multi-hop tool-call traces violate it routinely, and the violation is invisible at ingestion time — the attributes are well-formed, the span kinds are correct, and the trace is valid. The double count only materializes at query time, when a cost dashboard sums over the trace. ## What Was Observed A three-hop agent trace: `invoke_agent` (root) → `execute_tool` (hop 1) → `chat` (LLM call, hop 1) → `execute_tool` (hop 2) → `chat` (LLM call, hop 2). The agent framework's auto-instrumentation emitted: - Root `invoke_agent`: `gen_ai.usage.input_tokens = 4120`, `gen_ai.usage.output_tokens = 610` (cumulative across the run). - Each `chat` span: its own `gen_ai.usage.input_tokens` / `output_tokens` (per-call). Summing `gen_ai.usage.input_tokens` across the trace yields 4120 + per-call sums, roughly doubling the true billed input. The root's 4120 is the sum of the children's inputs plus the framework's own prompt assembly, so the trace contains the same tokens twice under two different span scopes. The failure is not detectable by schema validation, by the OTel collector's `genai` semantic-convention checks, or by span-status inspection. It is detectable only by a structural query: for each span carrying usage attributes, ask whether any descendant also carries usage attributes. A second, subtler variant appeared when the agent framework emitted *only* the root cumulative totals and instrumented the leaf `chat` spans with `gen_ai.operation.name` but no usage attributes (because the provider SDK did not report usage for a streamed call). Here the root totals were correct, but a per-call cost breakdown was impossible, and a naive "sum leaf usage" query returned zero. The two variants are mirror images: one over-counts, one under-counts, and both are structurally indistinguishable from a correct trace without inspecting span hierarchy. ## Condition of Validity The detection rule below is valid only when: 1. The trace is complete — all spans in the causal subtree are present in the query window, and the trace is not sampled such that a child `chat` span was dropped while its parent was kept. Head-based sampling that drops a child but keeps the parent produces a false negative (no double count detected) and a false positive (parent usage looks like a leaf). 2. `parent_span_id` links are intact. Some exporters emit `invoke_agent` and its `chat` children as siblings under a synthetic root when the framework's context propagation is broken; in that case the descendant relation is lost and the rule cannot fire. 3. The agent span's usage attributes are cumulative over its subtree, not a snapshot of a single internal call. Frameworks that reuse the agent span for the first LLM call only will *not* double count, and the rule will correctly not fire. It does **not** apply when usage is recorded exclusively at leaf spans and the agent span carries no usage attributes — the common, correct case. It also does not apply to `gen_ai.usage.*` on spans with `gen_ai.operation.name = "embeddings"`, which are leaf operations by definition. ## Decision Boundary When the rule fires, the operator must choose between two corrections, and the choice is not free: - **Drop the parent's usage attributes at ingestion** (rewrite the span before export). This restores summable leaf usage but destroys the agent run's total, which some consumers (per-run cost attribution, budget enforcement) require. It also mutates telemetry, which is a governance question. - **Tag the parent as aggregate and exclude it from leaf sums** (add a derived attribute such as `gen_ai.usage.aggregate = true` and require all sum queries to filter `WHERE NOT aggregate`). This preserves both views but pushes the invariant into every downstream query, where it will be forgotten. The boundary: if the organization has a single cost pipeline, rewriting is safe. If cost data is consumed by more than one team or by an external billing system, rewriting silently changes historical totals and the tag-and-filter approach is the only non-destructive path. The decision must be made before the first backfill, because retrofitting aggregate tags onto already-exported spans requires reprocessing raw span storage. ## Operational Effect Left undetected, the double count inflates cost dashboards by the agent's orchestration overhead factor — typically 1.4x to 2.1x for tool-heavy agents, since each hop re-sends accumulated context. The inflation is not constant: it scales with hop count, so it corrupts *relative* cost comparisons between shallow and deep agents more than absolute totals. Teams that tuned agent depth against a double-counted baseline systematically under-provisioned deep agents, because the marginal cost of each additional hop appeared higher than it was. The detection must run as a structural check on the trace, not on the span. The following guard is written to run inside a trace-assembly step, where the full span tree is available. ```python from __future__ import annotations from dataclasses import dataclass, field class TokenDoubleCountError(RuntimeError): """Raised when a span carries gen_ai.usage.* while a descendant also does.""" @dataclass class Span: span_id: str parent_span_id: str | None name: str operation: str | None # gen_ai.operation.name has_usage: bool # carries gen_ai.usage.input_tokens or output_tokens children: list["Span"] = field(default_factory=list) def _descendant_has_usage(span: Span) -> bool: stack = list(span.children) while stack: node = stack.pop() if node.has_usage: return True stack.extend(node.children) return False def assert_no_double_counted_usage(root: Span) -> None: """Walk the assembled trace and reject summable-usage violations. A span that carries usage attributes is only summable if no descendant span also carries them. Agent spans that accumulate child usage violate this invariant and must be tagged as aggregates before any cost query. """ stack = [root] while stack: span = stack.pop() if span.has_usage and _descendant_has_usage(span): raise TokenDoubleCountError( f"span {span.span_id!r} ({span.name!r}, " f"operation={span.operation!r}) carries gen_ai.usage.* " f"while a descendant also carries it; summing this trace " f"will double-count tokens. Tag the span as an aggregate " f"or strip its usage attributes before export." ) stack.extend(span.children) ``` Run this guard in the trace-assembly path, before any cost aggregation, and treat a raised `TokenDoubleCountError` as a telemetry defect to be fixed at the instrumentation source — not as a query-time filter to be worked around.