# Where must an instance reconcile GenAI agent spans against token-level telemetry: the tool-call boundary invariant

**Date of Record:** 2026 November 7

**Scope:** Applies to OpenTelemetry GenAI semantic conventions (v1.27+ `gen_ai.*` attributes) for agent spans that emit `gen_ai.operation.name = execute_tool` child spans alongside per-request token usage. Does not apply to single-shot completion spans without tool calls, nor to providers that report usage only at the parent `invoke_agent` span.

## Core Distinction

The GenAI semantic conventions define two orthogonal telemetry planes that share a trace but not a clock:

1. **Structural plane** — spans: `invoke_agent` → `chat` → `execute_tool` → `chat`, with attributes `gen_ai.tool.name`, `gen_ai.tool.call.id`, `gen_ai.operation.name`.
2. **Accounting plane** — token counters: `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, and (where supported) `gen_ai.usage.cache_read.input_tokens`, attached per `chat` span.

The non-obvious invariant is this: **the sum of `execute_tool` spans under an agent span must equal the count of tool-call IDs present in the `output_tokens` of the preceding `chat` span's response — and that equality is only checkable by correlating the tool-call ID set, not by counting spans.** Span counts and token counts do not reconcile directly. A model can emit two tool calls in one assistant turn (one `chat` span, two `execute_tool` children) or emit zero tool calls but a non-zero `output_tokens` delta for a text-only turn. The only reliable join key is `gen_ai.tool.call.id`, which must be propagated from the assistant message's `tool_calls[].id` into both the `execute_tool` span and the tool-result message fed back to the next `chat` span.

## What Was Observed

In a deployed agent runtime (LangGraph-style loop, OTel GenAI instrumentation v1.27), we observed three recurring divergences:

- **Orphaned `execute_tool` spans.** A tool call was retried at the transport layer after a 504. The retry produced a second `execute_tool` span with a *new* `gen_ai.tool.call.id` (the SDK generated one), while the assistant message retained the original ID. The parent `chat` span's `output_tokens` accounted for one tool call; the trace contained two `execute_tool` children. Token accounting and span accounting disagreed by exactly one call.
- **Phantom tool calls.** A model emitted a tool call in a streamed response that was truncated by a client-side `max_tokens` cutoff. The `chat` span recorded `output_tokens` for the partial emission (including the partial tool-call JSON), but the runtime never executed it, so no `execute_tool` span exists. The invariant is violated in the *opposite* direction: tokens spent, no span.
- **ID laundering at the proxy.** An LLM gateway rewrote `tool_calls[].id` (provider-format normalization, e.g., Anthropic `toolu_*` → OpenAI `call_*`) but did not rewrite the ID embedded in the already-emitted `execute_tool` span attributes. The join silently failed; the trace *looked* complete but the correlation graph was broken.

None of these produce an OTel error. All spans are `STATUS_CODE_OK`. The failure is purely in the reconciliation invariant.

## Condition of Validity

The equality `set(tool_call_ids in chat.output) == set(gen_ai.tool.call.id on execute_tool children)` holds **only if all of the following are true**:

- The `chat` span is emitted *after* the full assistant message is materialized, not at first-byte. Streaming instrumentation that closes the span on `finish_reason` is valid; instrumentation that closes on stream end without inspecting `tool_calls` is not.
- The runtime propagates the provider's tool-call ID verbatim through: assistant message → `execute_tool` span attribute → tool-result message. Any layer that regenerates or normalizes the ID must do so *before* the `chat` span closes.
- Retries reuse the original `gen_ai.tool.call.id`. A transport-level retry is the *same* logical tool call and must not mint a new ID.
- `gen_ai.usage.output_tokens` reflects the provider's reported count for the turn, including tokens spent on tool-call argument JSON. Providers that exclude tool-call tokens from `output_tokens` (some do) break the accounting half of the check but not the ID-set half.

If any condition fails, the invariant degrades from an equality to a *containment* check: every `execute_tool` span's ID must appear in some `chat` span's output tool-call set within the same trace, but the reverse is not guaranteed.

## Decision Boundary

The operational decision is **when to treat a mismatch as a hard error versus a degraded-telemetry warning**. Use this boundary:

- **Hard error (fail the trace's quality gate):** an `execute_tool` span exists whose `gen_ai.tool.call.id` appears in *no* `chat` span's output tool-call set within the trace. This means a tool executed that the model did not request in recorded telemetry — a correctness and security concern (possible prompt-injection-driven tool call, or ID laundering).
- **Soft warning (emit metric, do not fail):** a `chat` span's output tool-call set contains an ID with no corresponding `execute_tool` span. This is consistent with truncation, client abort, or a tool that was deliberately not executed (policy denial). Record as `genai.tool_call.unexecuted`.
- **Never reconcile on counts alone.** `count(execute_tool) == count(tool_calls)` is necessary but not sufficient; parallel tool calls and retries both break count-only checks while leaving the ID-set check intact.

## Operational Effect

Instrumenting the ID-set reconciliation as a span processor (not a tail sampler) catches the three observed failure modes at emit time. The practical consequences:

- Retry-minted IDs surface as hard errors and force the transport layer to reuse the originating ID — this is the fix, not a workaround.
- Truncated tool calls become visible as unexecuted-ID warnings, which correlate with `finish_reason = length` on the parent `chat` span.
- Gateway ID rewriting is caught because the `execute_tool` span carries the pre-rewrite ID; the reconciliation fails loudly rather than producing a plausible-looking but disconnected trace.

The cost is one span processor and a per-trace set join, negligible against the cost of debugging a silently broken agent trace after the fact.

```python
from __future__ import annotations

from dataclasses import dataclass, field


class ToolCallCorrelationError(RuntimeError):
    """Raised when execute_tool spans cannot be reconciled to chat-span tool-call IDs."""


@dataclass
class ChatSpan:
    span_id: str
    output_tool_call_ids: set[str] = field(default_factory=set)
    output_tokens: int = 0


@dataclass
class ExecuteToolSpan:
    span_id: str
    parent_chat_id: str
    tool_call_id: str


def reconcile_trace(
    chat_spans: list[ChatSpan],
    tool_spans: list[ExecuteToolSpan],
) -> dict[str, list[str]]:
    """Reconcile execute_tool spans against chat-span output tool-call IDs.

    Returns a dict of soft warnings keyed by kind. Raises
    ToolCallCorrelationError on hard violations (orphaned tool spans).
    """
    all_chat_ids: set[str] = set()
    for cs in chat_spans:
        all_chat_ids |= cs.output_tool_call_ids

    orphaned = [ts for ts in tool_spans if ts.tool_call_id not in all_chat_ids]
    if orphaned:
        # A tool executed that no recorded chat turn requested: hard failure.
        ids = sorted(ts.tool_call_id for ts in orphaned)
        raise ToolCallCorrelationError(
            f"orphaned execute_tool spans with no matching chat output: {ids}"
        )

    executed_ids = {ts.tool_call_id for ts in tool_spans}
    unexecuted = sorted(all_chat_ids - executed_ids)
    warnings: dict[str, list[str]] = {}
    if unexecuted:
        warnings["genai.tool_call.unexecuted"] = unexecuted

    # Count-only equality is necessary but not sufficient; record divergence
    # as a diagnostic without failing, since parallel calls make it expected.
    if len(tool_spans) != sum(len(cs.output_tool_call_ids) for cs in chat_spans):
        warnings["genai.tool_call.count_divergence"] = [
            f"spans={len(tool_spans)} ids={sum(len(cs.output_tool_call_ids) for cs in chat_spans)}"
        ]
    return warnings


def process_trace(chat_spans: list[ChatSpan], tool_spans: list[ExecuteToolSpan]) -> None:
    try:
        warnings = reconcile_trace(chat_spans, tool_spans)
    except ToolCallCorrelationError:
        # Re-raise with trace context attached for the span processor.
        raise
    for kind, entries in warnings.items():
        # Emit as metric/log; do not fail the trace.
        print(f"[warn] {kind}: {entries}")
```