# Which invariants prevent double-counting of tool-call spans under OpenTelemetry GenAI semantic conventions when agents delegate to sub-agents

**Date of Record:** 2026 November 8

**Scope:** Applies to traces where an LLM agent emits a tool call that is itself implemented as a nested agentic loop (sub-agent, planner, or "handoff") that performs its own tool calls, and where both layers are instrumented with `gen_ai.*` span attributes. Does not apply to flat single-agent tool loops, nor to traces where only one instrumentation layer is active (e.g., only the orchestrator or only the sub-agent runtime emits spans).

## Core Distinction

The OpenTelemetry GenAI semantic conventions define `gen_ai.operation.name` (values such as `chat`, `execute_tool`, `invoke_agent`, `create_agent`) and a set of usage attributes (`gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`) that are *per-inference-call*, not per-trace. The non-obvious boundary is the distinction between **a tool call as an operation** and **a tool call as the parent of a nested agent**. When an agent's tool is "delegate to sub-agent," the outer `execute_tool` span and the inner `invoke_agent` span both exist, both carry `gen_ai.tool.name`/`gen_ai.agent.name`, and both may carry usage attributes. The invariant that matters is that **usage attributes are additive only along disjoint inference boundaries**; a parent `execute_tool` span must not aggregate child inference usage into its own `gen_ai.usage.*`, because the child `chat` spans already report it. Violating this produces token counts that are correct per-span but 2–3× inflated at the trace root — the classic symptom being a trace whose root usage exceeds the sum of its leaf `chat` spans.

## What Was Observed

In a planner/executor setup where the planner calls a `research` tool that spins up a ReAct sub-agent, we saw root `gen_ai.usage.input_tokens` of 41,200 against a ground-truth of 13,900. Decomposition showed three contributors to the same tokens:

1. The sub-agent's three `chat` spans (leaf, correct: 13,900 total).
2. The `execute_tool` span for `research`, whose instrumentation copied the sub-agent's accumulated usage onto itself as a convenience for dashboards.
3. The outer `invoke_agent` span, whose instrumentation summed *all* descendant spans' usage attributes, including the already-aggregated `execute_tool` span.

This is a double-aggregation: leaf → tool span → agent span. The OTel spec does not forbid usage attributes on `execute_tool` spans, and many SDK wrappers populate them, so the failure is silent and spec-compliant. A second, subtler variant: when the sub-agent is a remote service reached over HTTP, the client-side `execute_tool` span and the server-side `invoke_agent` span share a `trace_id` via context propagation, and a naive collector-side "sum usage by trace" processor counts both the client's echoed usage and the server's authoritative usage.

We also observed `gen_ai.tool.call.id` collisions: the orchestrator generated a tool-call ID, and the sub-agent's runtime generated its own for the *same logical call*, so joining spans by `tool.call.id` produced a fan-out rather than a 1:1 link. The correct join key for delegation is parent-child span linkage (`parent_span_id`), with `gen_ai.tool.call.id` treated as opaque and scoped to the emitting layer.

## Condition of Validity

The "sum leaf `chat` usage" rule is valid only when:

- Every inference call is represented by exactly one span with `gen_ai.operation.name = chat` (or `text_completion`), and that span is a leaf with respect to other usage-bearing spans.
- `execute_tool` and `invoke_agent` spans are treated as **structural** (timing, linkage, names) and never as **usage-bearing**, or if they are usage-bearing, their usage is explicitly marked as an aggregate via a distinct attribute namespace (we use `app.aggregate.usage.*`, not `gen_ai.usage.*`).
- Context propagation across process boundaries is verified: if a remote sub-agent re-emits spans under the same trace, the client must not echo the server's usage.
- Streaming partial responses are not double-emitted as both a `chat` span and an `execute_tool` span for the same tokens.

When any of these fails, the leaf-sum is no longer the ground truth and a reconciliation pass is required.

## Decision Boundary

The decision is: **does the instrumentation layer own the token accounting, or does the semantic layer?** If a wrapper SDK populates `gen_ai.usage.*` on non-`chat` spans, that SDK has taken ownership and downstream aggregators must either (a) filter to `gen_ai.operation.name = chat` before summing, or (b) trust only spans flagged `gen_ai.usage.is_aggregate = false`. We chose (a): all trace-level usage is computed by a collector processor that selects leaf `chat` spans and sums them, ignoring any `gen_ai.usage.*` on `execute_tool`/`invoke_agent`. The boundary condition is that this processor must run *after* span-link resolution and *before* any sampling that could drop a leaf `chat` span while retaining its parent — tail-sampling by `error` status is safe, but head-sampling is not, because a dropped leaf silently undercounts.

## Operational Effect

After enforcing the leaf-only rule and moving aggregate usage into `app.aggregate.*`, root usage matched ground truth within 0.4% across 2,100 traces. The cost is that dashboards can no longer read `gen_ai.usage.*` off the root span directly; they must join to leaves. We accepted that cost because the alternative — trusting aggregate attributes — produced silent 3× billing-attribution errors that only surfaced during monthly reconciliation. The durable lesson: in multi-agent traces, `gen_ai.usage.*` is a *per-inference* fact, and any invariant that treats it as *per-trace* must be enforced by the consumer, not assumed from the producer, because the spec permits both and the producer is usually a third-party SDK you do not control.

```python
"""Guard: reject traces whose usage attributes violate the leaf-only invariant."""

from dataclasses import dataclass


class UsageInvariantError(RuntimeError):
    """Raised when gen_ai.usage.* appears on a non-leaf, non-chat span."""


@dataclass
class Span:
    span_id: str
    parent_span_id: str | None
    operation: str            # gen_ai.operation.name
    usage_input: int | None   # gen_ai.usage.input_tokens
    usage_output: int | None


def validate_usage_invariant(spans: list[Span]) -> int:
    """Return trace-level input tokens, or raise if the invariant is violated.

    Invariant: gen_ai.usage.* may only appear on spans with
    operation == 'chat' that have no usage-bearing descendants.
    """
    children: dict[str, list[Span]] = {}
    by_id = {s.span_id: s for s in spans}
    for s in spans:
        if s.parent_span_id:
            children.setdefault(s.parent_span_id, []).append(s)

    def has_usage(s: Span) -> bool:
        return s.usage_input is not None or s.usage_output is not None

    def usage_descendant(s: Span) -> bool:
        stack = list(children.get(s.span_id, []))
        while stack:
            cur = stack.pop()
            if has_usage(cur):
                return True
            stack.extend(children.get(cur.span_id, []))
        return False

    total = 0
    for s in spans:
        if not has_usage(s):
            continue
        if s.operation != "chat":
            raise UsageInvariantError(
                f"usage on non-chat span {s.span_id} "
                f"(operation={s.operation!r}); move to app.aggregate.* namespace"
            )
        if usage_descendant(s):
            raise UsageInvariantError(
                f"chat span {s.span_id} has usage-bearing descendant; "
                f"leaf-only invariant violated (double aggregation likely)"
            )
        total += s.usage_input or 0
    return total


def reconcile(trace_spans: list[Span], declared_root_input: int) -> None:
    computed = validate_usage_invariant(trace_spans)
    if declared_root_input and abs(computed - declared_root_input) / max(declared_root_input, 1) > 0.05:
        raise UsageInvariantError(
            f"root usage {declared_root_input} diverges from leaf sum {computed} "
            f"by >5%; trace is double-counted"
        )
```