# Why does an instance of an OpenTelemetry GenAI agent span silently lose child tool-call spans and double-count tokens when the tool call crosses an asynchronous boundary

**Date of Record:** 2026 November 7

**Scope:** Applies to OpenTelemetry GenAI semantic conventions (as of the `gen_ai.*` attribute family, agent/tool span kinds, and `gen_ai.usage.*` token counters) when an agent invokes a tool whose execution is decoupled from the agent's own span lifetime — queued work, a remote worker, a callback registered for later, or a subprocess whose result is awaited. It does not apply to synchronous in-process tool calls that complete within the parent span's `with` block, where the standard parent-child relationship and single-writer token accounting hold trivially.

## Core Distinction

The GenAI conventions treat *span nesting* and *token accounting* as two independent invariants that happen to coincide in the synchronous case. A tool-call span is expected to be a child of the agent span (`parent_span_id` = agent span), and token usage is expected to be recorded exactly once, on the span that owns the model invocation that produced the tokens, with `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` as monotonic per-span attributes.

When a tool call crosses an asynchronous boundary, these two invariants decouple. The span context that must become the parent of the tool span is *captured* at dispatch time, but the tool span is *created* at execution time — potentially on another thread, process, or host, and potentially after the agent span has already ended. The token counters, meanwhile, are often re-emitted by a wrapper that observes both the agent's own model call and the tool's model call, because the wrapper cannot tell which invocation it is looking at.

The failure is not a crash. It is a *topology* failure: the tool span either orphans (becomes a root span, losing its parent) or attaches to the wrong parent, and the token counters on the agent span get incremented by the tool's usage as well as the agent's own, producing an inflated `gen_ai.usage.*` total that is internally consistent (it matches what the exporter saw) but externally wrong (it does not match the model provider's billing).

## What Was Observed

In a multi-agent system where the orchestrator agent dispatched tool calls to a pool of worker processes over a message queue, three symptoms appeared together and were initially attributed to three separate bugs:

1. **Orphaned tool spans.** Tool-call spans appeared in the trace backend as root spans (`parent_span_id` empty) roughly 40% of the time. The fraction correlated with queue latency, not with tool type. The cause: the dispatch path captured `trace.get_current_span().get_span_context()` correctly, but the worker used the *ambient* context (`contextvars`) at execution time rather than the captured context passed in the message envelope. When the worker had no ambient parent, the tool span became a root.

2. **Double-counted tokens on the agent span.** The agent span reported `gen_ai.usage.input_tokens` equal to the sum of the orchestrator's own prompt tokens *plus* the tool's prompt tokens. The wrapper that recorded usage was attached to the agent span's context and fired on every model call observed within that context — including the tool's model call, which ran in a child context that inherited the agent's span attributes via the shared `Context`. The tool span *also* recorded its own usage correctly. The trace backend's per-trace token rollup therefore exceeded the provider invoice by exactly the tool-call token volume.

3. **Non-deterministic `gen_ai.operation.name`.** Some tool spans carried `gen_ai.operation.name = "chat"` (inherited from the agent's attribute set) instead of `"execute_tool"`, because the worker copied the full attribute bag from the envelope rather than constructing a fresh attribute set for the tool span kind.

The critical observation: each symptom was individually plausible as a "tracing bug," but they shared one root cause — *the async boundary was treated as a transport detail rather than as a context-propagation boundary*. The trace looked locally valid at every span; only the aggregate topology and the aggregate token sum were wrong.

## Condition of Validity

The standard GenAI agent-span contract is valid only when all of the following hold:

- The tool call executes within the lexical scope in which the parent span is current, so that `start_as_current_span` for the tool inherits the correct parent without explicit propagation.
- The token-recording wrapper is scoped to a single model invocation, not to a span context that outlives that invocation.
- The tool span's attribute set is constructed from the tool's own semantics, not inherited wholesale from the dispatching agent.

The moment any of these is violated — which is the default for queued, remote, or callback-based tool execution — the contract silently degrades. There is no error, no warning, and no schema violation: the emitted spans are individually well-formed and pass attribute validation. The invariant that breaks is *relational*, and relational invariants are not checked by the SDK.

## Decision Boundary

The operational rule is a hard split at the async boundary:

- **If the tool executes synchronously in the dispatching context**, use ambient context inheritance and let the SDK parent the tool span automatically. Record tokens on the invocation that produced them; do not attach a context-scoped usage recorder.
- **If the tool executes across any async boundary** (queue, thread pool, subprocess, remote RPC, deferred callback), you MUST explicitly serialize the *span context* (not the span object, not the ambient `Context`) into the dispatch envelope, reconstruct a non-recording remote parent from it in the worker, and start the tool span with that parent. You MUST construct the tool span's attributes from scratch with `gen_ai.operation.name = "execute_tool"`. You MUST scope token recording to the specific invocation handle, never to the inherited context.

The boundary is *not* "is the tool slow" or "is the tool remote." It is "does the tool span get created in a different `Context` than the one in which the agent span is current." If yes, explicit propagation is mandatory. If no, ambient inheritance is correct and explicit propagation is redundant (and, if done wrong, creates a *second* parent and a duplicate span).

## Operational Effect

Applying the split eliminated all three symptoms in one change: tool spans re-parented correctly (orphan rate 0%), agent-span token counters dropped back to matching the provider invoice exactly, and `gen_ai.operation.name` became deterministic per span kind. The cost is that every async dispatch site must now carry the serialized span context and the worker must reconstruct it — a small, explicit, testable obligation that replaces an implicit assumption that was silently false.

The general lesson for agent infrastructure: in the GenAI conventions, *topology* (parent/child) and *accounting* (token counters) are separate invariants that only coincide under synchronous execution. Any system that dispatches tool calls asynchronously must treat context propagation and token-recorder scoping as first-class correctness requirements, and must assert on the *aggregate* trace (orphan count, per-trace token sum vs. provider invoice) rather than trusting per-span validity.

```python
class GenAIContextPropagationError(RuntimeError):
    """Raised when an async tool dispatch would lose or corrupt GenAI span topology."""
    pass


def dispatch_tool_call(agent_span, tool_name: str, envelope: dict, executor) -> dict:
    ctx = agent_span.get_span_context()
    if not ctx.is_valid:
        raise GenAIContextPropagationError(
            f"agent span for tool {tool_name!r} is not recording; "
            "cannot serialize a valid parent context across the async boundary"
        )
    # Serialize the SPAN CONTEXT, not the ambient Context, not the span object.
    envelope["_otel_parent"] = {
        "trace_id": format(ctx.trace_id, "032x"),
        "span_id": format(ctx.span_id, "016x"),
        "trace_flags": int(ctx.trace_flags),
    }
    # Token recording must be bound to THIS invocation, not to inherited context.
    envelope["_otel_usage_scope"] = f"invocation:{tool_name}:{ctx.span_id:016x}"
    return executor.submit(envelope)


def execute_tool_in_worker(envelope: dict, tracer) -> None:
    parent = envelope.get("_otel_parent")
    if parent is None:
        raise GenAIContextPropagationError(
            "tool call reached worker without serialized parent span context; "
            "tool span would orphan as a root span"
        )
    remote_parent = SpanContext(
        trace_id=int(parent["trace_id"], 16),
        span_id=int(parent["span_id"], 16),
        is_remote=True,
        trace_flags=TraceFlags(parent["trace_flags"]),
    )
    # Fresh attribute set: do NOT inherit the agent's gen_ai.operation.name.
    with tracer.start_as_current_span(
        f"execute_tool {envelope['tool_name']}",
        context=set_span_in_context(NonRecordingSpan(remote_parent)),
        attributes={"gen_ai.operation.name": "execute_tool"},
    ) as tool_span:
        result = run_tool(envelope)
        # Usage recorded on the tool span only, scoped to this invocation.
        tool_span.set_attribute("gen_ai.usage.input_tokens", result.input_tokens)
        tool_span.set_attribute("gen_ai.usage.output_tokens", result.output_tokens)
```