# Which invariants prevent silent token and cost telemetry loss across OpenTelemetry GenAI agent spans **Date of Record:** 2026 November 16 **Scope:** Applies to multi-hop agent execution graphs where a parent `invoke_agent` span delegates to nested `execute_tool` / `chat` spans and where token usage and cost attributes are emitted per model invocation. Does not apply to single-shot LLM calls with no tool fan-out, nor to purely trace-only pipelines that never aggregate usage attributes for billing. ## Core Distinction The non-obvious boundary is between *span presence* and *attribute propagation*. The OpenTelemetry GenAI semantic conventions define attribute names (`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cost`, `gen_ai.operation.name`, `gen_ai.tool.name`, `gen_ai.tool.call.id`) but they do **not** define an aggregation invariant across the span tree. A trace can be perfectly complete — every agent, tool, and chat span present, correctly parented, correctly named — while the *sum of token usage observed at the root* is strictly less than the sum of tokens actually consumed. This is silent under-counting, not a crash. The failure mode is that token/cost attributes are leaf-attached facts. When an agent loop re-enters a model after a tool call, the second `chat` span is a sibling of the first under the agent span, not a child. If the collector's usage aggregator only walks `chat` spans that are direct children of the root, or only reads `gen_ai.usage` from spans where `gen_ai.operation.name == "chat"` *and* the parent is the root, the tool-mediated re-invocations vanish from the cost rollup. The trace looks healthy. The bill does not. ## What Was Observed In a ReAct-style agent with a planner, three tools, and a final synthesis step, the trace contained: 1 `invoke_agent` span, 3 `execute_tool` spans, and 4 `chat` spans (planner, two tool-followup reasoning turns, synthesis). Each `chat` span carried correct `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`. The root span carried `gen_ai.usage.input_tokens` equal to only the *first* chat turn's input tokens — because the instrumentation attached usage to the root at span start (a pre-flight estimate) and never reconciled it after child spans closed. Observed delta: root reported 1,842 input tokens; sum of `chat` children was 11,406. The 84% gap was not an error, not a dropped span, and not a sampling decision. It was a propagation invariant violation: the root's usage attribute was written exactly once, before the work that determined it. A second observation: tool spans carried `gen_ai.tool.name` but the `gen_ai.tool.call.id` on the `execute_tool` span did not match the `tool_call_id` embedded in the followup `chat` span's request messages. The correlation key existed on both sides but was not the same value, so any join-based cost attribution (tool cost → originating model turn) silently produced zero rows rather than failing. ## Condition of Validity The invariant holds — and must be enforced — only when all of the following are true: 1. The agent performs **iterative re-invocation** (tool-call loops, reflection, retries), so the number of model calls is not known at root-span start. 2. Usage attributes are aggregated for **billing or quota**, where under-counting has downstream consequences. 3. The collector or backend performs **structural aggregation** (walking the span tree) rather than trusting a single pre-computed total. If the agent is single-shot, or if usage is sourced from a provider-side billing API rather than span attributes, this boundary does not bind. The invariant is also *not* required when the root span is explicitly documented as "first-turn usage only" and a separate `gen_ai.usage.total_*` attribute is authoritative. ## Decision Boundary The correct decision is: **never write terminal usage attributes on a span before its children close.** Root-level usage must be a *derived* value computed at span end from the closed child set, or must be omitted from the root entirely and left to backend aggregation. The forbidden pattern is a root span that reports a usage number *and* has usage-bearing descendants — because consumers cannot distinguish "this is the total" from "this is a partial pre-estimate." Concretely, the boundary is: - If a span has any descendant with `gen_ai.usage.*`, then that span's own `gen_ai.usage.*` MUST be either absent or provably equal to the sum of descendants plus its own direct model calls. - Correlation keys (`gen_ai.tool.call.id`) MUST be byte-identical across the tool span and the referencing chat span; a "close enough" ID is a silent join failure, not a near-miss. - Cost (`gen_ai.usage.cost`) MUST NOT be derived from a token count that was itself derived from a partial aggregate — errors compound multiplicatively across the loop depth. ## Operational Effect When the invariant is enforced, cost rollups reconcile to provider invoices within rounding, and tool-attributed cost analysis is possible. When it is violated, the failure is invisible until invoice reconciliation, at which point the trace data cannot explain the gap because the trace is *structurally complete* — there is no missing span to point at. The remediation cost is high: retroactive re-aggregation requires re-reading every span in the window and recomputing, which is only possible if raw child spans were retained (i.e., if sampling did not drop them). This makes the invariant a *retention* dependency as well as a propagation one: you cannot recover a violated aggregate if the leaves were tail-sampled away. The practical guard is a validation pass at span-close time that rejects a root-span usage write when usage-bearing descendants exist, and a join-integrity check on tool-call IDs. The following enumerates the violation as a hard error rather than a metric. ```python from dataclasses import dataclass, field from typing import Optional class GenAIUsagePropagationError(RuntimeError): """Raised when a span reports usage that cannot be reconciled with its descendants.""" pass @dataclass class SpanRecord: name: str operation: str parent_id: Optional[str] span_id: str input_tokens: Optional[int] = None output_tokens: Optional[int] = None tool_call_id: Optional[str] = None referenced_tool_call_id: Optional[str] = None def validate_usage_invariant(root: SpanRecord, descendants: list[SpanRecord]) -> int: """Return authoritative total input tokens; raise if root usage is unreconcilable.""" usage_bearing = [ s for s in descendants if s.input_tokens is not None or s.output_tokens is not None ] descendant_input = sum(s.input_tokens or 0 for s in usage_bearing) if usage_bearing and root.input_tokens is not None: if root.input_tokens < descendant_input: raise GenAIUsagePropagationError( f"root span {root.span_id} reports {root.input_tokens} input tokens " f"but {len(usage_bearing)} descendants account for {descendant_input}; " f"root usage was written before children closed (silent under-count)" ) if root.input_tokens > descendant_input: raise GenAIUsagePropagationError( f"root span {root.span_id} reports {root.input_tokens} input tokens, " f"exceeding descendant sum {descendant_input}; uncorrelated usage" ) return descendant_input def validate_tool_call_joins(descendants: list[SpanRecord]) -> None: """Every referenced tool_call_id must resolve to exactly one tool span.""" tool_ids = {s.tool_call_id for s in descendants if s.operation == "execute_tool"} for s in descendants: if s.referenced_tool_call_id is not None: if s.referenced_tool_call_id not in tool_ids: raise GenAIUsagePropagationError( f"chat span {s.span_id} references tool_call_id " f"{s.referenced_tool_call_id!r} with no matching execute_tool span; " f"cost attribution join will silently produce zero rows" ) ```