# How does an instance detect token accounting drift when OpenTelemetry GenAI agent spans split a single model invocation across tool-call boundaries

**Date of Record:** 2026 November 10

**Scope:** Applies to services that emit `gen_ai.*` attributes on agent and tool spans per the OpenTelemetry GenAI semantic conventions, where a single logical model invocation is decomposed into a parent agent span plus one or more child tool-call spans. Does not apply to single-shot inference spans that carry no tool-call children, nor to providers that report a single aggregate `gen_ai.usage.total_tokens` with no prompt/completion breakdown.

## Core Distinction

The GenAI semantic conventions define `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` as attributes of the span that represents a *model invocation* — the call to the inference endpoint. Agent orchestration frameworks, however, frequently model a multi-turn tool-use loop as one logical "agent run" and emit a parent span (`gen_ai.operation.name = "invoke_agent"` or similar) plus child spans for each tool call, with a *nested* model invocation span per turn. The boundary that fails is not "did we record tokens" but **which span is authoritative for a given token count, and whether the sum across children is a partition or a duplication of the parent's total.**

A partition means the children's token counts sum to the parent's count because the parent is an aggregate. A duplication means the parent already carries the full multi-turn total while each child also carries its own per-turn total, so naive aggregation double-counts. The conventions do not mandate which pattern a producer uses; they only name the attributes. An instance that assumes one pattern will silently mis-bill or mis-rate-limit against the other.

## What Was Observed

Across three agent frameworks instrumented against the same backend, the same logical run (one user prompt, two tool calls, three model turns) produced three structurally different span trees:

1. **Flat aggregate.** One `invoke_agent` span carrying `gen_ai.usage.input_tokens = 4200`, `output_tokens = 310`, and *no* child model-invocation spans; tool calls appear as `execute_tool` sibling spans with no usage attributes. Summing usage over all spans yields the correct total.
2. **Nested per-turn.** Parent `invoke_agent` span carries *no* usage attributes; each of three child `chat` spans carries its own `input_tokens`/`output_tokens`. Summing over leaves yields the correct total; summing over all spans (including a parent that happens to also emit a cached copy) double-counts.
3. **Duplicated.** Parent `invoke_agent` span carries the full run total *and* each child `chat` span carries its per-turn total. Summing over all spans in the trace inflates the count by roughly the number of turns (here ~3x). This is the failure mode: it is internally consistent per-span, passes schema validation, and only reveals itself when compared against the provider's billing export.

The tell is not an error — it is a ratio. In the duplicated case, `sum(all span input_tokens) / provider_billed_input_tokens ≈ number_of_model_turns`. In the flat and nested cases the ratio is ≈1.0. An instance that never computes this ratio has no signal that its accounting is wrong; dashboards look plausible because the inflation is proportional and stable.

A second, subtler invariant breaks on tool-call spans specifically: `gen_ai.tool.call.id` must be unique per tool invocation and must match the `tool_call_id` echoed back in the *next* model turn's prompt. When an agent framework retries a tool call after a transient failure, it may emit two `execute_tool` spans with the same `gen_ai.tool.call.id` but only one of which the model actually consumed. Token accounting is unaffected, but tool-call *count* metrics and any cost model that prices tool calls independently will overcount. The retry span and the consumed span are indistinguishable by attributes alone; only the presence of the id in a subsequent prompt span's input disambiguates them.

## Condition of Validity

The drift-detection procedure below is valid only when all of the following hold:

- The trace contains at least one span with `gen_ai.operation.name` in the invocation set (`chat`, `text_completion`, `invoke_agent`) and at least one with `gen_ai.operation.name = "execute_tool"`.
- The provider exposes an independent billed-token figure (billing export, response header, or a side channel) for the same time window. Without an external reference, the ratio test cannot run; only cross-span consistency checks remain.
- Span parent/child relationships are intact — i.e., the collector did not drop parent spans or re-parent children during sampling. Tail-based sampling that keeps tool spans but drops the parent will make a nested tree look flat and invert the ratio.
- Token attributes are integers, not strings. Several SDKs emit `gen_ai.usage.input_tokens` as a string when the upstream API returns it as JSON string; summing strings concatenates rather than adds, producing absurd totals that mask the real duplication.

When the external reference is unavailable, the instance can still detect *structural* inconsistency: a parent span that carries usage attributes while also having children that carry usage attributes is a candidate for duplication, and should be flagged rather than summed.

## Decision Boundary

The instance must decide, per trace, whether the parent span is an **aggregate** or a **peer**:

- If the parent `invoke_agent` span has usage attributes **and** has child `chat` spans with usage attributes, treat the parent as authoritative and *exclude* children from the sum. Flag the trace as `duplication_suspected` until the ratio test confirms.
- If the parent has no usage attributes and children do, treat children as authoritative and sum over leaves only.
- If neither parent nor children carry usage, the trace is unaccounted; emit a `usage_missing` signal rather than treating the count as zero — zero is a valid count and must not be conflated with absence.

The boundary is crossed the moment the instance sums usage over *all* spans in a trace without first classifying the tree shape. That single line of aggregation logic is where flat, nested, and duplicated producers diverge, and it is the only place the divergence is observable before billing reconciliation.

## Operational Effect

An instance that classifies tree shape before summing recovers correct totals for all three producer patterns and, critically, emits a `duplication_suspected` signal on the duplicated pattern *before* the monthly billing variance appears. The ratio test then either clears the flag (ratio ≈ 1.0, producer is actually flat-with-copied-parent, harmless) or confirms it (ratio ≈ turn count, producer double-counts). In the confirmed case the fix is at the producer — drop usage from either parent or children — but the instance's job is to have *detected* it, not to silently correct it, because silent correction hides a producer bug that will resurface on the next framework upgrade.

The tool-call retry duplication is handled separately: dedupe `execute_tool` spans by `gen_ai.tool.call.id`, keeping the span whose id appears in a downstream prompt span's input; if no downstream reference exists (the run ended on the retry), keep the last-emitted span and mark the others `unconsumed_retry`. Tool-call count metrics computed without this dedupe overcount by the retry rate, which on flaky networks can be 5–15%.

## Code

```python
from __future__ import annotations

from dataclasses import dataclass
from typing import Iterable


class TokenAccountingDrift(RuntimeError):
    """Raised when span-level token sums cannot be reconciled to a billed total."""
    pass


@dataclass(frozen=True)
class Span:
    name: str
    operation: str            # gen_ai.operation.name
    input_tokens: int | None  # None == attribute absent, distinct from 0
    output_tokens: int | None
    parent_id: str | None
    tool_call_id: str | None = None


INVOCATION_OPS = {"chat", "text_completion", "invoke_agent"}
TOOL_OP = "execute_tool"


def classify_tree(spans: Iterable[Span]) -> str:
    spans = list(spans)
    parents = [s for s in spans if s.operation in INVOCATION_OPS and s.parent_id is None]
    children = [s for s in spans if s.operation in INVOCATION_OPS and s.parent_id is not None]
    parent_has_usage = any(
        p.input_tokens is not None or p.output_tokens is not None for p in parents
    )
    child_has_usage = any(
        c.input_tokens is not None or c.output_tokens is not None for c in children
    )
    if parent_has_usage and child_has_usage:
        return "duplicated_suspected"
    if parent_has_usage:
        return "flat_aggregate"
    if child_has_usage:
        return "nested_per_turn"
    return "unaccounted"


def authoritative_input_tokens(spans: Iterable[Span]) -> int:
    spans = list(spans)
    shape = classify_tree(spans)
    if shape == "flat_aggregate":
        return sum(
            s.input_tokens or 0
            for s in spans
            if s.operation in INVOCATION_OPS and s.parent_id is None
        )
    if shape == "nested_per_turn":
        return sum(
            s.input_tokens or 0
            for s in spans
            if s.operation in INVOCATION_OPS and s.parent_id is not None
        )
    if shape == "duplicated_suspected":
        # Cannot decide without external reference; refuse to guess.
        raise TokenAccountingDrift(
            "parent and child invocation spans both carry usage; "
            "shape=duplicated_suspected, reconcile against billed total before summing"
        )
    raise TokenAccountingDrift("no usage attributes on any invocation span; shape=unaccounted")


def reconcile(
    spans: Iterable[Span],
    billed_input_tokens: int,
    turn_count: int,
    tolerance: float = 0.02,
) -> float:
    """Return the observed/billed ratio; raise if it matches duplication signature."""
    spans = list(spans)
    try:
        observed = authoritative_input_tokens(spans)
    except TokenAccountingDrift:
        # Duplicated shape: fall back to parent-only sum for the ratio test.
        observed = sum(
            s.input_tokens or 0
            for s in spans
            if s.operation in INVOCATION_OPS and s.parent_id is None
        )
    if billed_input_tokens <= 0:
        raise TokenAccountingDrift("billed_input_tokens must be positive for ratio test")
    ratio = observed / billed_input_tokens
    if abs(ratio - 1.0) <= tolerance:
        return ratio
    if turn_count > 1 and abs(ratio - turn_count) / turn_count <= tolerance:
        raise TokenAccountingDrift(
            f"ratio={ratio:.3f} matches turn_count={turn_count}; "
            "producer double-counts usage across parent and child spans"
        )
    raise TokenAccountingDrift(
        f"ratio={ratio:.3f} unexplained; neither ~1.0 nor ~turn_count={turn_count}"
    )


def dedupe_tool_spans(spans: Iterable[Span], consumed_ids: set[str]) -> list[Span]:
    seen: dict[str, Span] = {}
    for s in spans:
        if s.operation != TOOL_OP or s.tool_call_id is None:
            continue
        if s.tool_call_id in consumed_ids:
            seen[s.tool_call_id] = s
        elif s.tool_call_id not in seen:
            seen[s.tool_call_id] = s  # keep last-emitted if never consumed
    return list(seen.values())
```