# How does an instance detect the Orphaned Tool-Call Span in OpenTelemetry GenAI Agent Hierarchies **Date of Record:** 2026 November 16 **Scope:** Applies to agents that emit `gen_ai.*` spans per the OpenTelemetry GenAI semantic conventions (agent → model → tool-call nesting) where tool execution is delegated to an out-of-process executor and token accounting is aggregated per agent. Does not apply to single-span synchronous inference calls, nor to deployments where tool execution is inlined within the parent model span's process. ## Core Distinction The GenAI semantic conventions model a tool call as a child span (`gen_ai.operation.name = "execute_tool"`) nested under the model-invocation span that requested it, with the model span carrying `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` and the tool span carrying `gen_ai.tool.name` and `gen_ai.tool.call.id`. The non-obvious invariant is this: **the tool-call span's parent must remain the model span that *issued* the call, not the agent span that *dispatched* it.** When an agent dispatches tool execution asynchronously, the natural implementation is to create the tool span from the dispatcher's context, which silently reparents it under the agent span. The span tree stays well-formed, passes schema validation, and every attribute is present — but the parent-child edge that token accounting and latency attribution depend on is now wrong. This is not a missing-span problem. It is a *wrong-parent* problem, and it is invisible to any validator that only checks span presence, attribute completeness, or tree acyclicity. ## What Was Observed In a multi-agent deployment, per-agent token totals drifted upward by 15–40% relative to the sum of model-span tokens, while the global total stayed correct. The cause: when a tool span is reparented to the agent span, the tool's own token-bearing sub-spans (e.g., a nested `gen_ai.chat` call made by the tool) attach under the *tool* span, which now sits under the *agent* rather than under the *model*. Aggregation that walks `agent → model → descendants` double-counts the tool's model calls because they are reachable via two paths — once through the agent's direct child (the reparented tool) and once through the model span that the tool logically belongs to but no longer descends from. The failure is *latent under load*. Under low concurrency the dispatcher's context and the model span's context are usually the same goroutine/task, so the reparenting is a no-op and the tree is correct. Under concurrency, the dispatcher runs on a pooled worker whose active span is the agent, and the reparenting becomes active. This is why the defect passed staging (serialized) and failed in production (parallel). A second, compounding observation: `gen_ai.tool.call.id` is unique per *invocation*, not per *span*. When a tool is retried after a transient executor failure, the retry reuses the same call id. Two spans share a call id, both reparented under the agent, and any deduplication keyed on call id collapses them — dropping the retry's tokens entirely. The drift therefore has both an inflationary mode (double-count) and a deflationary mode (retry collapse), and which one dominates depends on the retry rate. ## Condition of Validity The invariant "tool span parent = issuing model span" holds only when: 1. The tool span is created from a context that carries the model span as its active span — i.e., the model invocation must explicitly pass its span context into the dispatch boundary, not rely on ambient context propagation. 2. The tool executor does not itself create a new root or agent-scoped context. If the executor is a separate process, the model span's `traceparent` must be injected into the dispatch message and extracted on the executor side; otherwise the executor's local context (agent span) becomes the parent. 3. Retries preserve call-id uniqueness by appending an attempt suffix (e.g., `call.id#2`) or by emitting a distinct `gen_ai.tool.call.id` per attempt, so deduplication does not collapse distinct executions. The invariant does *not* hold — and the failure is expected, not a defect — when the tool is genuinely fire-and-forget and its output is not consumed by the model span. In that case the tool span legitimately belongs to the agent, and token attribution should exclude it from model-scoped totals. The distinction is whether the tool's result flows back into the model's context window. ## Decision Boundary Detect the reparenting by comparing, for each tool span, the span id of its parent against the span id recorded in the model span's `gen_ai.tool.call.id` → issuing-span index. If the tool span's parent is the agent span rather than the model span, and the tool's output token count is non-zero, the span is misparented and its tokens must be attributed to the model span, not aggregated at the agent level. The boundary condition that makes this a *decision* rather than a *check*: a tool span whose parent is the agent span is *correct* if and only if the tool made no model calls (zero token-bearing descendants). A tool span with token-bearing descendants under an agent parent is unambiguously the failure. A tool span with no token-bearing descendants under an agent parent is ambiguous — it may be a fire-and-forget tool (correct) or a misparented tool that simply didn't call a model (harmless). Do not flag the ambiguous case as an error; flag it as a *warning* and route it to the accounting reconciliation path rather than the span-rejection path. Reconciliation rule: `agent_total = sum(model_span tokens) + sum(agent-parented tool tokens where tool has no model descendants)`. Any tool span that violates the parent invariant *and* has model descendants is excluded from both terms and reported. ## Operational Effect Mis-detection has asymmetric cost. A false positive (flagging a fire-and-forget tool as misparented) inflates the reconciliation warning rate and trains operators to ignore the signal. A false negative (missing a reparented tool with model descendants) silently corrupts per-agent cost attribution — the failure mode that motivated this record, where per-agent budgets under-report and the agent is allowed to overspend by the drift margin before the global total triggers an alert. The global total is the last line of defense and it fires late, after the budget is already breached. The correct posture: treat the parent-invariant check as a *hard gate* on token aggregation (never aggregate a misparented tool's descendants into the agent total), and treat the ambiguity as a *soft signal* (warn, don't reject). The gate must run before aggregation, not after, because aggregation is where the double-count is materialized. ```python from dataclasses import dataclass, field from typing import Optional class MisparentedToolSpanError(RuntimeError): """Raised when a tool span with token-bearing descendants is parented under the agent span instead of the issuing model span.""" pass @dataclass class Span: span_id: str parent_id: Optional[str] operation: str # "agent" | "chat" | "execute_tool" tool_call_id: Optional[str] = None input_tokens: int = 0 output_tokens: int = 0 children: list = field(default_factory=list) def _has_model_descendants(span: Span) -> bool: stack = list(span.children) while stack: s = stack.pop() if s.operation == "chat" and (s.input_tokens + s.output_tokens) > 0: return True stack.extend(s.children) return False def assert_tool_parent_invariant( tool_span: Span, issuing_model_span_id: str, strict: bool = True, ) -> None: """ Verify the tool span is parented under the model span that issued it. - If parent == issuing model span: OK. - If parent != issuing model span but tool has NO model descendants: ambiguous (fire-and-forget or harmless misparent) -> warn only. - If parent != issuing model span AND tool HAS model descendants: hard failure; tokens would be double-counted. """ if tool_span.parent_id == issuing_model_span_id: return if not _has_model_descendants(tool_span): # Ambiguous: do not raise. Caller should log a warning and # route to reconciliation, not span rejection. return if strict: raise MisparentedToolSpanError( f"tool span {tool_span.span_id} (call_id={tool_span.tool_call_id}) " f"is parented under {tool_span.parent_id!r} but was issued by " f"model span {issuing_model_span_id!r}; tool has token-bearing " f"model descendants and would be double-counted in agent totals." ) def aggregate_agent_tokens( agent_span: Span, issuing_index: dict, # tool_call_id -> issuing model span_id strict: bool = True, ) -> int: """ Sum tokens attributable to the agent without double-counting reparented tool sub-trees. """ total = 0 for model_span in agent_span.children: if model_span.operation != "chat": continue total += model_span.input_tokens + model_span.output_tokens for tool_span in model_span.children: if tool_span.operation != "execute_tool": continue issuing = issuing_index.get(tool_span.tool_call_id) assert_tool_parent_invariant(tool_span, issuing, strict=strict) # tool descendants are already counted via model_span path; # do not re-add here. # Agent-parented tools: count only if they carry no model descendants. for tool_span in agent_span.children: if tool_span.operation != "execute_tool": continue if not _has_model_descendants(tool_span): total += tool_span.input_tokens + tool_span.output_tokens # else: excluded from both terms; reported via warning. return total ```