# What happens when an instance of a GenAI agent span is exported before its tool-call child spans are finalized

**Date of Record:** 2026 November 12

**Scope:** Applies to OpenTelemetry SDKs emitting GenAI semantic conventions (`gen_ai.*`) for agent workflows where model invocation spans and tool execution spans are produced by separate instrumentation layers (e.g., an LLM client wrapper and a tool-executor wrapper) and are exported through a batching processor. Does not apply to synchronous single-threaded instrumentation where the parent span is guaranteed to end after all children, nor to exporters that buffer entire traces until root completion (e.g., certain tail-sampling collectors).

## Core Distinction

The GenAI semantic conventions define two distinct span shapes: the **model invocation span** (`gen_ai.operation.name = "chat"` / `"text_completion"`, carrying `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`) and the **tool execution span** (`gen_ai.operation.name = "execute_tool"`, carrying `gen_ai.tool.name`, `gen_ai.tool.call.id`). In an agent loop these are siblings under an agent-invocation span, but they are frequently emitted by *different* instrumentation libraries with *different* lifecycle boundaries.

The non-obvious distinction: `gen_ai.tool.call.id` on the tool span is the join key back to the `tool_calls[].id` embedded in the model invocation's assistant message, but that join is only reconstructable **if both spans land in the same trace and the parent agent span has not already been flushed with a terminal status**. When the model invocation span ends and is batched for export before the tool span that consumes its output has ended, the collector receives a parent whose `end_time` precedes its child's `start_time` — a temporal inversion that most backends silently accept but that breaks any downstream reconstruction of the agent reasoning chain.

## What Was Observed

In a LangChain-style agent runner instrumented with the OpenTelemetry GenAI conventions, the agent invocation span was being ended by the orchestrator as soon as the model returned a `tool_calls` payload, on the assumption that "the model call is done." The tool executor then ran asynchronously and emitted its `execute_tool` span with `parent_span_id` pointing at the already-ended agent span.

Symptoms in the exported data:

1. **Orphaned-by-timestamp tool spans.** The tool span's `start_time_unix_nano` was consistently 40–900 ms *after* the parent's `end_time_unix_nano`. Jaeger rendered these as detached; Tempo dropped them from trace-detail views unless queried by trace ID directly.
2. **Lost `gen_ai.usage.*` correlation.** Because the model invocation span (which owns token counts) and the tool span (which owns the call ID) were finalized at different times, span-metric processors (`spanmetrics` connector) aggregated them into separate metric streams keyed by `gen_ai.operation.name`, with no shared trace-level attributes. Per-tool token attribution became impossible.
3. **Event ordering inversion.** `gen_ai.tool.message` events attached to the agent span (per the events convention) arrived at the collector *before* the `gen_ai.tool.call.id` attribute on the tool span, so event-to-span joins on call ID failed at ingest.
4. **Batching processor reordering.** The `BatchSpanProcessor` flushed the parent on its 5 s schedule while the tool span sat in a different processor's queue; on high-load runs the parent arrived at the collector up to 4.2 s before the child, and a downstream OTLP→Parquet sink that assumed parent-before-child wrote the child into a later partition.

## Condition of Validity

This failure requires the conjunction of:

- **Separate instrumentation scopes** for model invocation and tool execution (the common case with framework-level auto-instrumentation plus a user-supplied tool wrapper).
- **Parent span ended by the orchestrator before tool dispatch completes** — i.e., the agent span's lifetime is bounded by the model call, not the full agent turn.
- **A batching or queuing exporter** between SDK and collector, such that export order is not guaranteed to match end order.
- **A backend or sink that relies on parent/child temporal ordering or on co-arrival for span joining** (span-metric connectors, trace-to-dataset pipelines, event-to-span correlators).

If any one is absent — e.g., a synchronous tool executor, or a tail-sampling collector that holds the trace until root end — the inversion does not manifest.

## Decision Boundary

Do **not** end the agent-invocation span at model-return when the response contains `tool_calls`. The correct boundary is: the agent span ends only after every tool span it spawned has ended and its result has been appended to the conversation. Concretely:

- Treat the model invocation span and each tool span as children of a single agent-turn span whose lifetime spans `model → tool → model` round trips.
- Propagate `gen_ai.tool.call.id` as a span attribute on the tool span *at start*, not at end, so event correlation survives reordering.
- If the orchestrator cannot be restructured to hold the parent open (e.g., a streaming API that yields tool calls incrementally), emit an explicit `gen_ai.agent.turn.id` resource attribute shared by all spans in the turn and accept that parent/child ordering is not guaranteed — then configure the collector to join on that key rather than on span parentage.

The invariant to enforce: **for every `execute_tool` span, there must exist an ancestor `chat` span with `end_time >= child.end_time`, or an explicit turn-correlation attribute must be present.** Violation is a schema-mapping defect, not a backend quirk.

## Operational Effect

After restructuring the agent runner to hold the turn span open across tool execution and to stamp `gen_ai.tool.call.id` at span start, the observed effects were:

- Tool spans stopped appearing as orphans; trace-detail views showed the full `chat → execute_tool → chat` chain in 100% of sampled traces (previously ~62%).
- The `spanmetrics` connector produced a single `gen_ai.client.token.usage` series per agent turn, restoring per-tool token attribution.
- Event-to-span joins on `gen_ai.tool.call.id` succeeded at ingest rate, eliminating a downstream reprocessing job that had been backfilling ~1.4M events/day.
- The Parquet sink's parent-before-child assumption held; partition skew between parent and child rows dropped to zero.

The residual cost is that agent-turn spans are now long-lived (seconds to minutes for multi-step agents), which increases in-memory span pressure on the SDK and delays visibility of the turn in live views. This is the correct trade: a delayed-but-consistent trace is strictly better than a fast-but-unjoinable one for any consumer that reconstructs agent behavior.

```python
from __future__ import annotations

from dataclasses import dataclass


class GenAiSpanOrderingError(RuntimeError):
    """Raised when a GenAI span tree violates the parent/child temporal invariant."""


@dataclass(frozen=True)
class SpanRecord:
    name: str
    operation_name: str  # gen_ai.operation.name
    span_id: str
    parent_span_id: str | None
    start_ns: int
    end_ns: int
    tool_call_id: str | None = None
    turn_id: str | None = None


def validate_agent_turn(spans: list[SpanRecord]) -> None:
    """Enforce: every execute_tool span has an ancestor chat span ending after it,
    or carries an explicit turn-correlation attribute."""
    by_id = {s.span_id: s for s in spans}

    def ancestors(span: SpanRecord):
        cur = span
        while cur.parent_span_id is not None:
            parent = by_id.get(cur.parent_span_id)
            if parent is None:
                return
            yield parent
            cur = parent

    for span in spans:
        if span.operation_name != "execute_tool":
            continue

        # Tool call id must be present at start for event correlation.
        if span.tool_call_id is None:
            raise GenAiSpanOrderingError(
                f"execute_tool span {span.span_id} missing gen_ai.tool.call.id; "
                "event-to-span joins will fail under reordering"
            )

        chat_ancestor = next(
            (a for a in ancestors(span) if a.operation_name == "chat"), None
        )
        if chat_ancestor is None:
            if span.turn_id is None:
                raise GenAiSpanOrderingError(
                    f"execute_tool span {span.span_id} has no chat ancestor and no "
                    "gen_ai.agent.turn.id; trace is unjoinable"
                )
            continue

        if chat_ancestor.end_ns < span.end_ns:
            raise GenAiSpanOrderingError(
                f"temporal inversion: chat span {chat_ancestor.span_id} ends at "
                f"{chat_ancestor.end_ns} before tool span {span.span_id} ends at "
                f"{span.end_ns}; agent turn span was closed at model-return"
            )
```