# What happens when an instance treats gen_ai.usage.input_tokens as inclusive of cached tokens under OpenTelemetry GenAI semantic conventions **Date of Record:** 2026 November 17 **Scope:** Applies to services emitting `gen_ai.*` span attributes per OpenTelemetry GenAI semantic conventions (v1.27+) where prompt caching or context reuse is enabled on the upstream model provider. Does not apply to providers that report uncached input tokens only, nor to spans that omit usage attributes entirely. ## Core Distinction The GenAI semantic conventions define `gen_ai.usage.input_tokens` as the number of tokens in the prompt, but they do not normatively constrain whether that count includes tokens served from a provider-side cache. Providers diverge: some report `input_tokens` as the *total* prompt size with a separate `cache_read_input_tokens` sub-count, others report `input_tokens` as *only* the uncached remainder. A span that aggregates usage across heterogeneous providers, or that computes cost from `input_tokens` alone, will silently mis-account unless it knows which convention each backend follows. The boundary failure is not a crash — it is a plausible-looking, systematically wrong token ledger that passes schema validation and skews routing and billing decisions downstream. ## What Was Observed A router fanning out to two backends emitted spans with identical attribute names but divergent semantics: - Backend A (cache-reported-inclusive): `gen_ai.usage.input_tokens = 8400`, `gen_ai.usage.cache_read.input_tokens = 8000`, `gen_ai.usage.output_tokens = 220`. True billed prompt tokens: 400 new + 8000 cache-read (priced differently). - Backend B (cache-reported-exclusive): `gen_ai.usage.input_tokens = 380`, `gen_ai.usage.output_tokens = 210`. The cache hit was surfaced only in a non-standard `x-cache-hit-tokens` header that the collector dropped. Any aggregation that sums `input_tokens` across both spans reports 8780 prompt tokens. The true *new* prompt tokens were 400 + 380 = 780. The error is an order of magnitude, and it is invisible in the span itself because both spans are individually valid. The failure surfaces only when a downstream cost model, a context-window budgeter, or a routing policy compares spans across backends. We observed a routing policy that began preferring Backend B because its "prompt tokens per request" looked 20x smaller — an artifact of the accounting convention, not of efficiency. The policy then overloaded Backend B's cache-miss path and raised end-to-end latency by 31%. A second observation: `gen_ai.usage.output_tokens` is comparatively safe because reasoning/thinking tokens are usually folded in by convention, but tool-call argument tokens are *not* consistently counted. A span with a large `gen_ai.tool.call` payload may show `output_tokens` that excludes the serialized arguments, so a token-budget guard keyed on output_tokens under-predicts context growth for tool-heavy agents. ## Condition of Validity The invariant "`input_tokens` equals billed prompt tokens" holds only when the emitting backend reports uncached-only input and no cache-read sub-attribute is present. It is violated whenever `gen_ai.usage.cache_read.input_tokens` (or a provider-specific equivalent) is present and non-zero, because the parent `input_tokens` may then be either inclusive or exclusive depending on backend. The invariant "`output_tokens` covers all generated content" holds only for backends that count tool-call argument serialization; it is violated for tool-heavy turns on backends that report only natural-language completion tokens. Neither invariant can be validated from a single span — both require out-of-band knowledge of the backend's reporting contract, which must be attached at emission time (e.g., as a resource attribute identifying the provider and its accounting mode) rather than inferred at query time. ## Decision Boundary Treat a GenAI span's usage attributes as **non-comparable across providers** unless a `gen_ai.provider.name` (or equivalent) resource attribute plus a known accounting mode is present. The decision boundary is: if `cache_read.input_tokens` is absent, do not assume cache-exclusive accounting — the backend may simply not report cache at all, which is indistinguishable from a cache miss. Route on *derived* metrics (new prompt tokens = `input_tokens - cache_read.input_tokens` when inclusive, or `input_tokens` when exclusive, gated on a declared mode), never on raw `input_tokens`. Reject the temptation to normalize at query time by guessing; a wrong guess yields a confident, wrong ledger. ## Operational Effect The practical consequence is that token accounting and cost attribution must be a *typed* operation with an explicit accounting mode per backend, and the emitter must refuse to produce a usage record it cannot classify. The code below enforces this at the emission boundary: it raises rather than emitting an ambiguous or internally inconsistent usage record, so the collector never ingests a span whose token semantics are unknown. ```python from dataclasses import dataclass from enum import Enum class AccountingMode(Enum): CACHE_INCLUSIVE = "inclusive" # input_tokens includes cache_read tokens CACHE_EXCLUSIVE = "exclusive" # input_tokens is uncached remainder only NO_CACHE_REPORTED = "none" # backend reports no cache breakdown class AmbiguousUsageError(RuntimeError): pass @dataclass(frozen=True) class GenAIUsage: input_tokens: int output_tokens: int cache_read_input_tokens: int | None mode: AccountingMode provider: str def new_prompt_tokens(self) -> int: if self.mode is AccountingMode.CACHE_INCLUSIVE: if self.cache_read_input_tokens is None: raise AmbiguousUsageError( f"{self.provider}: inclusive mode requires cache_read_input_tokens" ) if self.cache_read_input_tokens > self.input_tokens: raise AmbiguousUsageError( f"{self.provider}: cache_read {self.cache_read_input_tokens} " f"exceeds input_tokens {self.input_tokens}" ) return self.input_tokens - self.cache_read_input_tokens if self.mode is AccountingMode.CACHE_EXCLUSIVE: if self.cache_read_input_tokens not in (None, 0): raise AmbiguousUsageError( f"{self.provider}: exclusive mode must not report cache_read" ) return self.input_tokens # NO_CACHE_REPORTED: cannot distinguish miss from unreported hit. raise AmbiguousUsageError( f"{self.provider}: no cache breakdown; new-prompt count is unknowable" ) def emit_usage_span_attributes(usage: GenAIUsage) -> dict[str, int | str]: # Validate before emission so the collector never sees an ambiguous record. new_tokens = usage.new_prompt_tokens() # raises AmbiguousUsageError if invalid attrs: dict[str, int | str] = { "gen_ai.usage.input_tokens": usage.input_tokens, "gen_ai.usage.output_tokens": usage.output_tokens, "gen_ai.usage.accounting_mode": usage.mode.value, "gen_ai.usage.new_prompt_tokens": new_tokens, "gen_ai.provider.name": usage.provider, } if usage.cache_read_input_tokens is not None: attrs["gen_ai.usage.cache_read.input_tokens"] = usage.cache_read_input_tokens return attrs def route_by_prompt_cost(spans: list[GenAIUsage]) -> str: # Routing must use new_prompt_tokens, never raw input_tokens. totals: dict[str, int] = {} for u in spans: totals[u.provider] = totals.get(u.provider, 0) + u.new_prompt_tokens() if not totals: raise AmbiguousUsageError("no comparable usage records to route on") return min(totals, key=totals.get) ```