# In what scenarios must an instance treat OpenTelemetry GenAI agent spans as non-authoritative for token accounting when tool calls cross process boundaries **Date of Record:** 2026 November 15 **Scope:** Applies to agent runtimes emitting OpenTelemetry GenAI semantic conventions (`gen_ai.*`) where an agent span orchestrates tool calls that are executed by a separate process, service, or model provider. Does not apply to single-process, single-provider LLM calls where the span and the token usage originate from the same SDK invocation. ## Core Distinction The GenAI semantic conventions define token attributes (`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, and the newer `gen_ai.usage.total_tokens`) as *per-inference* measurements attached to the span that represents that inference. An **agent span** (a span with `gen_ai.operation.name` of `invoke_agent` or equivalent orchestration role) is a *parent* span that aggregates child inference spans and tool-execution spans. The non-obvious boundary is this: **an agent span's token totals are a derived rollup, not an authoritative measurement.** When tool calls fan out to sub-agents, remote tool servers, or nested provider calls, the child spans may be emitted by a different process with a different resource, different sampling decision, and potentially a different clock. The parent agent span cannot reliably reconstruct the true token cost by summing child attributes, because child spans may be dropped by tail-sampling, may report usage under a different convention version, or may not report usage at all (tool servers that call an LLM internally without emitting `gen_ai.usage.*`). The distinction that matters operationally: **span-level token attributes are authoritative only at the span that directly wraps the provider response.** Any aggregation upward — especially across a process boundary — is an estimate and must be labeled as such. ## What Was Observed In a multi-agent deployment, an orchestrator agent span emitted `gen_ai.usage.input_tokens = 1204` and `gen_ai.usage.output_tokens = 88` reflecting only its own planning inference. It spawned three tool calls, two of which were executed by remote tool servers that each performed an internal LLM call to synthesize a response. Those internal calls emitted their own spans with correct token attributes, but the tool servers were configured with a head-based sampler at 10% and had a *different* `service.name`, so the child spans were frequently absent from the collector. The orchestrator's rollup logic summed visible children and produced a total that undercounted true token spend by 40–70% depending on sampling luck. A second failure mode appeared simultaneously: the remote tool servers were on an older semantic-conventions version emitting `gen_ai.usage.prompt_tokens` / `gen_ai.usage.completion_tokens` (pre-stabilization naming) rather than `input_tokens` / `output_tokens`. The rollup code read only the new names, silently treating legacy-named usage as zero. The agent span therefore *looked* complete — it had token attributes, it had children — but the numbers were structurally wrong, not merely sampled. A third, subtler condition: when a tool call failed and the agent retried, the retry produced a new child inference span, but the failed attempt's tokens were still billed by the provider. The failed span was often not exported (error spans dropped by an exporter filter), so retry-inflated cost was invisible in the trace while present in the invoice. ## Condition of Validity Treat an agent span's token attributes as authoritative **only if all of the following hold**: 1. Every child inference span is emitted by the same process and resource as the agent span (no cross-process tool execution that itself invokes a model). 2. All child spans use the same semantic-conventions version for usage attribute names, or the rollup explicitly normalizes both `input_tokens`/`prompt_tokens` and `output_tokens`/`completion_tokens`. 3. The sampling configuration guarantees child spans are not independently dropped relative to the parent (e.g., parent-based sampling with `ParentBased(AlwaysOn)` for the agent subtree). 4. Retried and failed inference attempts are exported, not filtered, so their token cost is counted. If any condition fails, the agent span's totals are a **lower bound with unknown error**, and downstream billing, quota enforcement, or capacity planning must not consume them as ground truth. ## Decision Boundary The decision boundary is *where the rollup is consumed*. For **observability dashboards** (latency, rough cost trends), sampled rollups are acceptable — label them `estimated=true`. For **billing reconciliation, per-tenant quota metering, or cost attribution**, the agent-span rollup must be rejected in favor of provider-side usage records or a dedicated metering path that is independent of trace sampling. The boundary is crossed the moment a consumer requires exactness; at that point the trace-derived number is disqualified regardless of how complete it appears. A practical rule: if the agent span's token total is used to make a decision that has a financial or access-control consequence, and any tool call in its subtree crossed a process boundary, the number is invalid for that decision. ## Operational Effect Teams that ignored this boundary shipped quota systems that under-billed heavy users by the sampling gap and over-served tenants whose tool calls happened to be sampled out. The fix was to split accounting: trace spans carry `estimated=true` rollups for debugging, while a separate synchronous metering callback at each provider call site records exact usage to a durable ledger. The agent span remained useful for *correlating* cost to a workflow, but ceased to be the source of truth for *quantifying* it. The invariant to encode: **never derive a billing-grade number by summing attributes across a span boundary you do not control.** ```python from dataclasses import dataclass from typing import Iterable class NonAuthoritativeTokenRollupError(RuntimeError): """Raised when an agent-span token rollup is consumed as billing-grade truth despite crossing a process/sampling/convention boundary.""" @dataclass class SpanUsage: span_id: str service_name: str usage_convention: str # "stable" | "legacy" input_tokens: int output_tokens: int sampled: bool def assert_rollup_is_authoritative( agent_service: str, children: Iterable[SpanUsage], require_exact: bool, ) -> int: """Validate that an agent span's token rollup may be trusted. Returns the total token count only when the rollup is authoritative. Raises NonAuthoritativeTokenRollupError otherwise. """ total = 0 for child in children: if child.service_name != agent_service: if require_exact: raise NonAuthoritativeTokenRollupError( f"child span {child.span_id} from service " f"'{child.service_name}' crosses a process boundary; " f"rollup is not billing-grade" ) if child.usage_convention != "stable": if require_exact: raise NonAuthoritativeTokenRollupError( f"child span {child.span_id} uses legacy usage convention " f"'{child.usage_convention}'; names do not match stable " f"gen_ai.usage.input_tokens" ) if not child.sampled: if require_exact: raise NonAuthoritativeTokenRollupError( f"child span {child.span_id} was not sampled; rollup is a " f"lower bound with unknown error" ) total += child.input_tokens + child.output_tokens return total def meter_for_billing(agent_service: str, children: list[SpanUsage]) -> int: # Billing path demands exactness; any boundary invalidates the rollup. return assert_rollup_is_authoritative(agent_service, children, require_exact=True) ```