# Why does an instance of the OpenTelemetry GenAI agent span silently lose tool-call attribution when the tool invocation outlives the enclosing agent span **Date of Record:** 2026 November 17 **Scope:** Applies to instrumentations emitting `gen_ai.*` spans per the OpenTelemetry GenAI semantic conventions where an agent span (`gen_ai.operation.name = "invoke_agent"`) encloses tool executions (`gen_ai.operation.name = "execute_tool"`). Does not apply to synchronous in-process tool calls that complete before the agent span ends, nor to backends that reconstruct causality from a separate trace propagation mechanism. ## Core Distinction The GenAI conventions define tool-call telemetry along two axes: spans (`execute_tool`) and attributes on the *agent* span summarizing the turn (`gen_ai.agent.tool_calls`, token counts, finish reasons). These are two distinct records of the same event, and they are joined only by parent-child span relationships and by attribute snapshots taken at span *end* time. The failure mode is not "missing data." It is *split attribution*: the tool span exists and is correct, but the agent span's aggregate attributes are frozen before the tool completes, so the aggregate and the child disagree. Downstream consumers that trust the agent-level rollup (cost dashboards, tool-success-rate SLIs, agent trajectory evaluators) see a tool call that never happened, or a success rate that excludes the slowest and most failure-prone invocations. ## What Was Observed Consider an agent span opened at turn start, with `gen_ai.agent.tool_calls` and `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` set as the model streams and requests tools. The agent then dispatches a tool call that is asynchronous or long-running (a web fetch, a code-execution sandbox, a human-in-the-loop approval). Two orderings occur in practice: 1. **Buffered/exported-at-end attributes.** Many SDKs snapshot span attributes at `span.end()`. If the agent span's `end()` is called when the model's turn concludes — which, for streaming agents, can be *before* the dispatched tool returns — the tool-call attribute reflects only tools known at that instant. The later-arriving `execute_tool` span becomes a child of an already-ended parent. OTel permits this (a child may end after its parent), but the parent's attributes are immutable post-end. Result: `gen_ai.agent.tool_calls` = 0 or 1 while two `execute_tool` spans exist as children. 2. **Token accounting skew.** `gen_ai.usage.output_tokens` on the agent span is finalized at turn end. If the tool's result is fed back into a *second* model call that is parented to the same agent span (a multi-step ReAct loop collapsed into one agent span), the second call's tokens are attributed to a different span or dropped entirely, and the agent-level token total undercounts the actual billed usage. The discrepancy is invisible unless you sum child `gen_ai.usage.*` and compare to the parent. In both cases the trace is structurally valid — no exporter error, no dropped span — which is why this escapes validation. A span-tree validator sees a well-formed parent-child edge. Only a *semantic* check (parent attribute vs. child count, parent tokens vs. child token sum) catches it. ## Condition of Validity The invariant "the agent span's `gen_ai.agent.tool_calls` equals the number of `execute_tool` children, and `gen_ai.usage.*` equals the sum over model-call children" holds **only if** every tool dispatched by the agent completes before the agent span ends, and every model call in the turn is parented to the agent span and ends before it. It is violated whenever: - Tool execution is asynchronous relative to the agent's turn boundary (fire-and-forget dispatch, streaming tool results, deferred approval). - The agent span is ended at model-turn end rather than at agent-loop end. - A retry or follow-up model call is emitted under a *new* agent span but the tool result is attributed to the original. The invariant is *not* restored by trace-context propagation alone; propagation fixes span linkage, not attribute finalization. ## Decision Boundary The operational choice is where to close the agent span: - **Close at model-turn end** (common default): low latency to export, but aggregate attributes are provably incomplete for async tools. Acceptable only if no consumer reads agent-level `tool_calls`/`usage` as authoritative. - **Close at agent-loop end** (after all dispatched tools return and all follow-up model calls complete): preserves the aggregate invariant, but the agent span's duration now includes tool wall-clock, inflating any latency SLI keyed on `invoke_agent` duration. - **Hybrid**: close the agent span at loop end for the aggregate, but emit a distinct `execute_tool` span with explicit `gen_ai.tool.name`, `gen_ai.tool.call.id`, and a `tool.result.status`, and make the *tool span* — not the agent rollup — the source of truth for tool SLIs. This is the only configuration that keeps both the aggregate invariant and latency semantics correct, at the cost of two sources of truth that must be reconciled. Choose the hybrid when tool success rate or per-tool latency is an SLI. Choose loop-end closure when agent-level token/cost rollups are the primary consumer. Never choose turn-end closure while also exposing `gen_ai.agent.tool_calls` to dashboards — that is the configuration that produces the silent undercount. ## Operational Effect The concrete effect is a stable, low-noise bias: fast tools are counted, slow and failing tools are systematically excluded from agent-level aggregates. Tool success rate computed from the agent rollup trends *upward* as failures get slower; cost dashboards undercount exactly the turns that ran the most tools. Because the bias is deterministic and correlated with tool latency, it does not average out across a fleet — it compounds. The mitigation is a reconciliation check at ingestion: assert parent-child consistency and reject or flag agent spans whose aggregates disagree with their children. ```python from dataclasses import dataclass, field from typing import Optional class AgentTelemetryInvariantError(RuntimeError): """Raised when an agent span's aggregate GenAI attributes disagree with its children.""" pass @dataclass class SpanRecord: name: str operation: str # gen_ai.operation.name parent_id: Optional[str] span_id: str ended: bool = True attributes: dict = field(default_factory=dict) def reconcile_agent_span(agent: SpanRecord, children: list[SpanRecord]) -> None: if agent.operation != "invoke_agent": return # invariant only defined for agent spans tool_children = [c for c in children if c.operation == "execute_tool"] model_children = [ c for c in children if c.operation in ("chat", "text_completion") ] declared_tools = agent.attributes.get("gen_ai.agent.tool_calls") if declared_tools is not None and declared_tools != len(tool_children): raise AgentTelemetryInvariantError( f"agent span {agent.span_id}: gen_ai.agent.tool_calls=" f"{declared_tools} but {len(tool_children)} execute_tool children " f"present; agent span likely closed before async tool returned" ) declared_in = agent.attributes.get("gen_ai.usage.input_tokens") declared_out = agent.attributes.get("gen_ai.usage.output_tokens") if declared_in is not None and declared_out is not None: child_in = sum(c.attributes.get("gen_ai.usage.input_tokens", 0) for c in model_children) child_out = sum(c.attributes.get("gen_ai.usage.output_tokens", 0) for c in model_children) if declared_in != child_in or declared_out != child_out: raise AgentTelemetryInvariantError( f"agent span {agent.span_id}: usage mismatch " f"(parent in/out={declared_in}/{declared_out}, " f"children sum in/out={child_in}/{child_out}); " f"a follow-up model call may be parented elsewhere" ) for tool in tool_children: if tool.ended and not agent.ended: continue if tool.ended and agent.ended and tool.span_id not in agent.attributes.get( "_reconciled_children", [] ): # child ended after parent: structurally legal, semantically suspect raise AgentTelemetryInvariantError( f"tool span {tool.span_id} ended after parent agent span " f"{agent.span_id}; aggregate attributes cannot include it" ) ```