# In what scenarios must an instance treat per-turn token accounting as non-additive across trace spans in GenAI observability pipelines **Date of Record:** 2026 November 15 **Scope:** Applies to observability pipelines that reconstruct per-agent-turn cost and safety attribution by summing token counts across child spans of a single trace root, where the LLM client (SDK, gateway, or proxy) emits usage records at more than one layer. Does not apply to pipelines that ingest a single authoritative usage event per turn from a billing-grade source (e.g., provider-side usage API keyed to a request ID), nor to single-span traces. ## Core Distinction The naive mental model is that a trace is a tree of spans, each span reports `input_tokens` and `output_tokens`, and the per-turn cost is the root-to-leaf sum. This holds only when exactly one layer in the call stack owns the tokenizer accounting and all other layers pass through usage verbatim or omit it. The failure boundary is the **double-counting invariant**: a turn's token cost is additive across spans *if and only if* the set of spans carrying usage records is an antichain in the causal tree with respect to token ownership — that is, no usage-bearing span is an ancestor of another usage-bearing span within the same request. When a gateway span wraps a provider-client span, and both emit `gen_ai.usage.input_tokens` (per the OpenTelemetry GenAI semantic conventions), the sum is inflated by the inner layer's count. When a retry loop re-issues the same prompt across sibling spans, the sum is inflated by the number of attempts, but only the successful attempt's tokens are billable *and* only the failed attempts' tokens are attributable to reliability cost — two different numbers that must not be conflated. This is non-obvious because both inflation modes produce *plausible* totals. Nothing throws. The dashboard shows a number. The number is wrong in a direction that is invisible without a reconciliation oracle. ## What Was Observed In a pipeline correlating three signal classes per turn — (1) trace spans with usage attributes, (2) a token-economics rollup table keyed by `turn_id`, and (3) safety-signal events emitted by a guardrail sidecar tagged with `turn_id` and `span_id` — the following divergence appeared under load: - **Sum-of-spans exceeded provider invoice by 1.6× to 2.3×** on turns routed through the LLM gateway, because the gateway emitted its own usage span *and* the underlying SDK span beneath it emitted usage. The ratio tracked the fraction of traffic routed through the gateway, not any property of the prompts. - **Safety events orphaned from cost.** Guardrail events carried the `span_id` of the *inner* SDK span, but the economics rollup keyed on the *outer* gateway span. Joins on `span_id` dropped ~40% of safety events on gateway-routed turns, silently. The orphan rate was zero on direct-SDK turns, which masked the bug during canary. - **Retry storms distorted per-turn safety attribution.** A turn that failed a content filter, retried with a mutated prompt, and succeeded produced two safety events (one block, one pass) under one `turn_id`. Downstream, the "blocked" event was counted as a turn-level outcome, inflating the block rate by the retry multiplier. The correct unit was the *attempt*, not the turn. - **Streaming turns under-counted output tokens** when the client span closed before the final usage chunk arrived; the gateway span, which awaited the full stream, was the only span with complete `output_tokens`. Here the outer span was *more* correct, inverting the usual "prefer the innermost" heuristic. The unifying observation: **there is no fixed layer that is authoritative for usage.** Authority is a property of the (provider, transport, streaming mode, retry policy) tuple, not of tree depth. ## Condition of Validity The additive-sum reconstruction is valid when all of the following hold for a given turn: 1. Exactly one span in the turn's subtree carries a usage record, **or** all usage-bearing spans are declared, via an explicit `usage_authority` attribute, to be non-overlapping partitions of the request (e.g., a fan-out to N models where each child is a distinct billable call). 2. Every safety-signal event's `span_id` resolves to a span in the same trace *and* the economics rollup joins on the trace's designated `usage_authority` span, not on an arbitrary span. 3. Retries are modeled as sibling attempt-spans under a `turn` span, with the turn span carrying no usage of its own, so that `sum(attempts) = turn_cost` is a deliberate identity rather than an accident. 4. For streaming turns, the span chosen as authority is the one whose lifetime strictly contains the terminal usage chunk — typically the outermost client that drains the stream to completion. When any condition is unmet, the pipeline must not sum. It must select an authority and record the selection. ## Decision Boundary The operational rule is a **precedence function over usage-bearing spans**, not a fold: - If a span declares `usage_authority=exclusive`, it wins and all other usage on the subtree is recorded as `shadow_usage` (retained for reconciliation, excluded from cost). - If no span declares authority, prefer the span whose transport observed the terminal byte of the response (streaming-aware), and emit a `usage_authority_inferred` warning. - If two spans both declare `exclusive`, raise — this is a configuration defect, not a data condition. - Safety events attach to the **attempt** span, and the turn-level safety outcome is a reduction over attempts (`blocked_any`, `blocked_final`, `retry_after_block`), never a sum. - Any turn where `sum_of_usage_spans` and `authority_span_usage` differ by more than a tolerance must be flagged for reconciliation; the divergence itself is the signal that authority inference was wrong. The boundary is crossed — and the pipeline must refuse to emit a cost figure — when authority is ambiguous *and* the divergence exceeds tolerance. Emitting a number here is worse than emitting none, because the number is consumed by budget enforcement and safety-rate dashboards that have no independent oracle. ## Operational Effect After enforcing authority selection: gateway-routed invoice reconciliation converged to within 0.4% (residual attributable to provider-side rounding on cached-input tokens). Safety-event orphan rate on gateway traffic dropped from ~40% to 0.1%, with the remainder traced to spans dropped by the collector's tail-sampling policy — a separate, now-visible defect. Retry-inflated block rates corrected by the measured retry multiplier, changing the reported block rate by a factor of 1.9 on the affected route. Streaming output-token under-count was eliminated by moving authority to the stream-draining span. The durable lesson: in multi-layer GenAI stacks, token usage is **not a conserved quantity across spans** — it is a per-layer observation of the same underlying billing event, and the observability pipeline's job is to *elect* an authority, not to *sum* observations. Pipelines that conflate the two produce stable, plausible, and wrong numbers that survive every smoke test because no smoke test reconciles against the invoice. ```python from __future__ import annotations from dataclasses import dataclass, field from enum import Enum from typing import Iterable class AmbiguousUsageAuthorityError(RuntimeError): """Raised when a turn's token cost cannot be attributed to a single authority span.""" class Authority(Enum): EXPLICIT = "explicit" INFERRED = "inferred" @dataclass(frozen=True) class UsageSpan: span_id: str parent_id: str | None input_tokens: int output_tokens: int declares_exclusive: bool = False drains_stream: bool = False @property def total(self) -> int: return self.input_tokens + self.output_tokens @dataclass(frozen=True) class TurnCost: turn_id: str authority_span_id: str authority: Authority billable_tokens: int shadow_tokens: int divergence: float warnings: tuple[str, ...] = field(default_factory=tuple) TOLERANCE = 0.02 # 2% divergence between summed observations and elected authority def resolve_turn_cost( turn_id: str, spans: Iterable[UsageSpan], *, tolerance: float = TOLERANCE ) -> TurnCost: spans = list(spans) if not spans: raise AmbiguousUsageAuthorityError( f"turn {turn_id!r}: no usage-bearing spans; cannot elect authority" ) explicit = [s for s in spans if s.declares_exclusive] warnings: list[str] = [] if len(explicit) > 1: raise AmbiguousUsageAuthorityError( f"turn {turn_id!r}: multiple spans declare usage_authority=exclusive: " f"{[s.span_id for s in explicit]}" ) if explicit: authority_span, authority = explicit[0], Authority.EXPLICIT else: stream_drainers = [s for s in spans if s.drains_stream] if len(stream_drainers) == 1: authority_span, authority = stream_drainers[0], Authority.INFERRED warnings.append("usage_authority_inferred:selected_stream_drainer") elif len(stream_drainers) > 1: raise AmbiguousUsageAuthorityError( f"turn {turn_id!r}: multiple stream-draining spans, no declared authority: " f"{[s.span_id for s in stream_drainers]}" ) else: authority_span, authority = spans[-1], Authority.INFERRED warnings.append("usage_authority_inferred:selected_last_span") summed = sum(s.total for s in spans) billable = authority_span.total shadow = summed - billable divergence = (shadow / billable) if billable else float("inf") if divergence > tolerance: raise AmbiguousUsageAuthorityError( f"turn {turn_id!r}: authority span {authority_span.span_id!r} reports " f"{billable} tokens but summed observations are {summed} " f"(divergence {divergence:.1%} > tolerance {tolerance:.1%}); " f"refusing to emit cost figure" ) return TurnCost( turn_id=turn_id, authority_span_id=authority_span.span_id, authority=authority, billable_tokens=billable, shadow_tokens=shadow, divergence=divergence, warnings=tuple(warnings), ) ```