# Which invariants prevent double-counting and span-nesting corruption in OpenTelemetry GenAI agent spans under retry and fan-out

**Date of Record:** 2026 November 14

**Scope:** Applies to instrumentation that emits `gen_ai.*` spans per the OpenTelemetry GenAI semantic conventions for agents, tool calls, and token usage, specifically when an agent framework performs internal retries, parallel tool fan-out, or nested sub-agent delegation. Does not apply to single-shot, non-retrying LLM calls where one span maps 1:1 to one provider request.

## Core Distinction

The GenAI semantic conventions describe *what* to record (`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.operation.name`, `gen_ai.tool.name`, `gen_ai.tool.call.id`) but they do not, by themselves, define *where the accounting boundary sits*. The non-obvious failure is a conflation of two different questions:

1. **Attribution** — which span "owns" a token count (a billing/telemetry question).
2. **Aggregation** — how a parent span's totals relate to its children (a rollup question).

The invariant that matters is: **token usage attributes are leaf-local, never rolled up.** A `gen_ai.usage.*` attribute on a span MUST describe only the model invocation that span directly represents. It is *not* a subtree total. Any consumer that sums `gen_ai.usage.input_tokens` across all spans in a trace without respecting span hierarchy will double-count on every nested agent or retried call. This is the boundary that silently corrupts dashboards.

## What Was Observed

A three-level agent (orchestrator → sub-agent → tool-using LLM) with a retry wrapper produced the following in a single trace:

- The orchestrator span carried `gen_ai.usage.input_tokens = 4200`.
- The sub-agent span carried `gen_ai.usage.input_tokens = 4200` as well — because the framework's "aggregate usage" helper copied child totals onto the parent.
- The leaf LLM span carried the true `gen_ai.usage.input_tokens = 1400`.

A naive `sum(gen_ai.usage.input_tokens)` over the trace yielded 9800 for a request that actually consumed 1400 input tokens. Separately, a retried tool call emitted **two** `execute_tool` spans sharing the same `gen_ai.tool.call.id`. Downstream dedup logic keyed on call ID collapsed them, but the retry's latency and the failed attempt's error status were then lost — the surviving span reported success with the *first* attempt's timestamps.

The critical observation: **both corruptions are invisible at span-emission time.** Each individual span is schema-valid. The invariant is violated only in the *relationship* between spans, which no single span can enforce.

## Condition of Validity

The leaf-local invariant holds — and rollups are safe — only when all of the following are true:

- Every span that carries `gen_ai.usage.*` corresponds to exactly one provider request/response pair. Retries MUST emit a **new span per attempt**, and only the successful (or final) attempt's usage is the "billable" one; failed attempts may carry usage but MUST be marked with `error.type` / non-OK status so aggregators can exclude them.
- Parent spans MUST NOT copy child `gen_ai.usage.*` values onto themselves. If a subtree total is needed, it is computed by the *consumer* via hierarchy-aware rollup, not stamped by the producer.
- `gen_ai.tool.call.id` MUST be stable across retries of the *same logical* call, while a distinct `span_id` distinguishes attempts. Dedup must key on `(tool.call.id, attempt)` — never on `tool.call.id` alone.
- `gen_ai.operation.name` MUST differ between `chat`, `execute_tool`, and `invoke_agent` so that a rollup can select the correct leaf level without guessing from span names.

When any of these is violated — particularly the parent-copies-child pattern — the trace remains structurally valid OTel but is semantically unusable for cost accounting.

## Decision Boundary

Choose the accounting strategy based on whether your pipeline can see span hierarchy:

- **If the consumer has full trace trees** (e.g., a backend with parent/child linkage): emit leaf-local usage only and compute rollups at query time. This is the correct default.
- **If the consumer sees only a flat span stream** (e.g., a metrics exporter that converts spans to counters): you MUST tag each usage-bearing span with an explicit role — `gen_ai.agent.role = leaf` vs `aggregate` — and the exporter MUST filter to `leaf` before incrementing counters. Do not rely on `operation.name` alone; a mislabeled sub-agent will leak.
- **If retries are involved:** never emit usage on the wrapper span. Emit per-attempt spans, and let the aggregator sum only `status == OK` attempts, or explicitly carry `gen_ai.usage.input_tokens` on failed attempts with `error.type` set so they are excluded from billing but retained for diagnostics.

The boundary is crossed the moment a producer stamps a subtree total onto a span that also has children carrying their own usage. At that point no downstream consumer can recover the truth without knowing your framework's private aggregation rule.

## Operational Effect

Teams that violate the leaf-local invariant typically discover it via a cost dashboard that reads 3–7× the provider invoice, or via p99 latency that is *lower* than reality because retried attempts were deduplicated away. The failure is silent, schema-passing, and only surfaces under retry or fan-out — i.e., exactly the conditions that matter most for agent reliability. Enforcing the invariant at the instrumentation layer (a guard that rejects usage attributes on spans with children, and that rejects duplicate `(tool.call.id, attempt)` pairs) converts a downstream accounting corruption into an immediate, local, actionable error at emit time.

```python
from __future__ import annotations

from dataclasses import dataclass, field


class GenAiSpanInvariantError(RuntimeError):
    """Raised when a GenAI span violates leaf-local accounting or call-id invariants."""


@dataclass
class GenAiSpan:
    span_id: str
    operation_name: str  # "chat" | "execute_tool" | "invoke_agent"
    parent_id: str | None = None
    input_tokens: int | None = None
    output_tokens: int | None = None
    tool_call_id: str | None = None
    attempt: int = 1
    status_ok: bool = True
    children: list["GenAiSpan"] = field(default_factory=list)


def validate_trace(spans: list[GenAiSpan]) -> None:
    by_id = {s.span_id: s for s in spans}

    # Rule 1: usage attributes are leaf-local; a span with children must not
    # carry gen_ai.usage.* (prevents double-counting on rollup).
    for span in spans:
        if span.children and (span.input_tokens is not None or span.output_tokens is not None):
            raise GenAiSpanInvariantError(
                f"span {span.span_id} ({span.operation_name}) carries usage attributes "
                f"but has {len(span.children)} children; usage must be leaf-local"
            )

    # Rule 2: tool-call identity is (call_id, attempt); duplicates corrupt dedup.
    seen: set[tuple[str, int]] = set()
    for span in spans:
        if span.operation_name == "execute_tool":
            if span.tool_call_id is None:
                raise GenAiSpanInvariantError(
                    f"execute_tool span {span.span_id} missing gen_ai.tool.call.id"
                )
            key = (span.tool_call_id, span.attempt)
            if key in seen:
                raise GenAiSpanInvariantError(
                    f"duplicate tool attempt {key}: retries must increment 'attempt' "
                    f"while preserving tool.call.id"
                )
            seen.add(key)

    # Rule 3: parent linkage must resolve.
    for span in spans:
        if span.parent_id is not None and span.parent_id not in by_id:
            raise GenAiSpanInvariantError(
                f"span {span.span_id} references unknown parent {span.parent_id}"
            )


def rollup_input_tokens(spans: list[GenAiSpan]) -> int:
    """Hierarchy-aware rollup: sum only leaf, successful attempts."""
    validate_trace(spans)
    total = 0
    for span in spans:
        if span.children:
            continue
        if not span.status_ok:
            continue
        total += span.input_tokens or 0
    return total
```