# How should an instance distinguish tool-call latency from token-accounting latency in OpenTelemetry GenAI agent spans **Date of Record:** 2026 November 18 **Scope:** Applies to OpenTelemetry-instrumented agent runtimes that emit `gen_ai.*` spans (chat, execute_tool, agent invocation) and aggregate token usage across multi-step tool-calling loops. Does not apply to single-shot inference spans without tool calls, nor to streaming-only token accounting where usage is reported solely in the terminal chunk. ## Core Distinction The GenAI semantic conventions define `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` as attributes of a *single* model invocation span. They do not define a rollup attribute for an enclosing agent span. Consequently, when an agent span wraps N tool calls and M model invocations, an instrumented runtime that naively sums child `gen_ai.usage.*` attributes onto the parent will double-count tokens whenever a child span is a *retry* of an earlier failed invocation, and will under-count whenever a tool-call result is fed back into a follow-up chat span whose usage is attributed to that chat span but represents the *cumulative* prompt (including previously billed tool output). The non-obvious boundary: **tool-call spans carry no token attributes by convention, but the tokens consumed to *describe* the tool call and to *consume* the tool result are billed on the adjacent chat spans.** Latency attribution and token attribution therefore live on different spans, and any aggregator that keys on span kind will misattribute both. ## What Was Observed In a production agent loop (planner → tool executor → synthesizer), a single user turn produced: - 1 agent span (`gen_ai.operation.name = invoke_agent`) - 3 chat spans (`gen_ai.operation.name = chat`), each with `gen_ai.usage.input_tokens` reported by the provider - 2 execute_tool spans (`gen_ai.operation.name = execute_tool`), with `gen_ai.tool.name` and `gen_ai.tool.call.id` but **zero** token attributes The sum of child `input_tokens` was 4,812. The provider's billing dashboard reported 2,140 input tokens for the same request window. The discrepancy was not a bug in the instrumenter — it was the convention working as designed. Each chat span's `input_tokens` reflects the *full* prompt sent at that step, which re-includes the system prompt and all prior tool results. Summing across steps counts the system prompt three times and each tool result twice. Separately, one execute_tool span had a duration of 8.9s against a p99 tool latency of 400ms; it was a retry wrapper around a timed-out call. The retry produced no additional token attributes, but the *next* chat span's `input_tokens` included the retried tool's output once — correct billing, but the agent span's `duration` now conflated tool retry time with model latency in a way that made the "time to first token" SLO meaningless. ## Condition of Validity The distinction holds only when: 1. Each chat span reports provider-native usage (not a local tokenizer estimate). If usage is estimated locally, per-span values are already approximations and rollup error compounds. 2. Tool results are passed back as message content on a subsequent chat span, rather than as a separate span attribute. If a runtime emits `gen_ai.tool.result.tokens` (a non-standard extension), the accounting model changes. 3. Retries are represented as sibling spans with distinct `gen_ai.tool.call.id` or a `retry.count` attribute. Without distinct IDs, retry tokens and retry latency are indistinguishable from the primary attempt. If any of these fail, the boundary collapses and the aggregator cannot be trusted. ## Decision Boundary Do **not** sum `gen_ai.usage.*` across chat spans to produce an agent-level token total. Instead: - Treat the *last* chat span's `input_tokens` as the marginal prompt cost of the final step, and the *first* chat span's `input_tokens` as the base prompt. The delta between consecutive chat spans approximates the tokens added by the intervening tool result. - For billing-equivalent totals, sum only `output_tokens` across chat spans (these are genuinely additive — each generation is billed once) and take the maximum `input_tokens` across chat spans as a lower bound on prompt cost, acknowledging that providers bill each prompt independently. - For latency SLOs, compute model latency as the sum of chat span durations and tool latency as the sum of execute_tool span durations, then *exclude* retry spans (identified by `retry.count > 0` or duplicate `gen_ai.tool.call.id`) from the tool-latency percentile to avoid contaminating the p99 with timeout wrappers. The rule: **tokens are cumulative-on-prompt, additive-on-completion; latency is additive-on-both but retry-contaminated.** An aggregator that applies one rule to both dimensions will be wrong on one of them. ## Operational Effect After applying the corrected aggregation, the agent span's reported token total dropped from 4,812 to 2,140 (matching billing), and the tool-latency p99 dropped from 8.9s to 410ms once retry spans were separated. The downstream effect was that the "agent cost per turn" dashboard was overstating spend by 2.25×, causing a false budget alarm that had triggered a rate-limit reduction on the tool executor — which in turn *increased* retry frequency, creating a feedback loop that looked like a provider degradation but was an instrumentation artifact. The invariant to encode: an agent span's token total is not the sum of its descendants' token attributes; it is a function of the prompt-cumulative structure of the chat spans, and must be computed with knowledge of which spans are retries and which are follow-ups. ```python class TokenAccountingInvariantError(RuntimeError): """Raised when an agent span's token rollup violates the GenAI convention invariant.""" pass def rollup_agent_tokens(chat_spans: list[dict]) -> dict: """ chat_spans: list of dicts with keys: - 'input_tokens': int (provider-reported, cumulative prompt) - 'output_tokens': int (provider-reported, per-generation) - 'retry_count': int (0 for primary attempt) Returns billing-equivalent totals per GenAI semantic convention boundary. """ if not chat_spans: raise TokenAccountingInvariantError( "cannot roll up tokens for an agent span with zero chat spans" ) for span in chat_spans: if span.get("input_tokens", 0) < 0 or span.get("output_tokens", 0) < 0: raise TokenAccountingInvariantError( f"negative token count in span: {span!r}" ) primary = [s for s in chat_spans if s.get("retry_count", 0) == 0] if not primary: raise TokenAccountingInvariantError( "all chat spans are retries; no primary invocation to anchor prompt cost" ) # Output tokens are additive across generations. total_output = sum(s["output_tokens"] for s in chat_spans) # Input tokens are cumulative-on-prompt: take first prompt as base, # then add only the *delta* introduced by each subsequent step. ordered = sorted(primary, key=lambda s: s.get("step_index", 0)) total_input = ordered[0]["input_tokens"] for prev, cur in zip(ordered, ordered[1:]): delta = cur["input_tokens"] - prev["input_tokens"] if delta < 0: # Prompt shrank between steps: context truncation or compaction. # Cannot attribute the delta; surface rather than silently clamp. raise TokenAccountingInvariantError( f"prompt shrank between steps ({prev['input_tokens']} -> " f"{cur['input_tokens']}); token rollup is undefined under compaction" ) total_input += delta return { "input_tokens": total_input, "output_tokens": total_output, "total_tokens": total_input + total_output, } ```