# What happens when an instance of a GenAI agent span is closed before its tool-call child spans: the OpenTelemetry GenAI semantic-convention invariant that tool-call spans must never outlive their enclosing agent invocation

**Date of Record:** 2026 November 14

**Scope:** Applies to traces emitted under the OpenTelemetry GenAI semantic conventions (`gen_ai.*`) where an agent loop (`gen_ai.operation.name` = `invoke_agent` or `chat`) emits tool-call spans (`execute_tool`) as descendants. Does not apply to flat, single-shot LLM inference spans with no tool execution, nor to non-GenAI spans that merely carry a `gen_ai.*` attribute for correlation.

## Core Distinction

The GenAI semantic conventions model an agent loop as a *parent* span (`invoke_agent`) that encloses zero or more *chat* spans (`chat`), each of which may enclose one or more *tool execution* spans (`execute_tool`). The non-obvious invariant is **temporal nesting, not just structural nesting**: the `execute_tool` child span must both *start after* and *end before* its enclosing `invoke_agent` span ends. The OpenTelemetry SDK will happily emit a child span whose `end_time` is later than its parent's `end_time` — the wire format permits it, the collector accepts it, and most backends will render it. But the resulting trace is semantically invalid under the GenAI conventions, and downstream consumers that reconstruct the agent loop from span timestamps (rather than from `parent_span_id`) will compute a negative or truncated tool-call duration and misattribute the tool latency to the *next* agent turn.

The distinction that matters operationally: **structural parentage (`parent_span_id`) is enforced by the SDK; temporal containment (`start_time`/`end_time` within the parent's interval) is not.** The GenAI conventions require both. A trace can be structurally a tree and temporally a mess.

## What Was Observed

In a production agent runtime using an async tool executor, the following sequence reproduced a class of broken traces:

1. The `invoke_agent` span was started with `start_as_current_span("invoke_agent")` and entered via context manager.
2. The agent issued a `chat` span, received a tool-call request, and dispatched the tool as an `asyncio.create_task(...)` so the loop could continue.
3. The tool task captured the *current* context (including the `invoke_agent` span as parent) at creation time — correct parentage.
4. The agent loop hit its token budget, returned a final answer, and the `invoke_agent` context manager exited, calling `span.end()` and stamping `end_time = T_agent_end`.
5. The tool task was still running. It completed at `T_tool_end > T_agent_end`, called its own `span.end()`, and the OTLP exporter shipped a span with `parent_span_id` pointing at a span that had already ended.

The resulting trace, viewed in a backend that sorts spans by `start_time`, showed the `execute_tool` span *after* the agent's final `chat` span — visually a sibling of the next request's spans. When the trace was re-ingested into an evaluation pipeline that computed "tool latency within agent turn" as `min(tool.end, agent.end) - tool.start`, the value was negative for 3.2% of tool calls, and those calls were silently dropped from the eval set. The agent's own logs reported the tool succeeded; the trace said the tool never ran. The divergence was invisible until the eval pipeline's coverage metric dropped.

The root cause was not the async dispatch per se — it was that the `invoke_agent` span was ended by the *loop's* control flow while tool tasks were still in flight, with no `asyncio.gather` on outstanding tool futures before `span.end()`.

## Condition of Validity

The invariant "every `execute_tool` span is temporally contained in its enclosing `invoke_agent` span" holds if and only if the agent span is ended *after* all tool-call futures it spawned have resolved. Concretely, the agent loop must:

- Track every tool-call task it creates (e.g., a `set[asyncio.Task]`), and
- `await asyncio.gather(*outstanding, return_exceptions=True)` (or equivalent) *before* exiting the `invoke_agent` context manager, and
- Ensure the `execute_tool` span's `end()` is called in a `finally` block on the tool task, not deferred to a callback that may fire after the agent span ends.

The invariant is **violated** whenever the agent span's `end_time` precedes any descendant `execute_tool` span's `end_time`. It is also violated, less obviously, when the tool span is created with `context=ctx` captured at dispatch time but the *agent* span is ended by a different coroutine — the parentage is correct but the temporal window is not.

## Decision Boundary

When you detect a tool span ending after its agent parent, the correct remediation depends on which of two regimes you are in:

- **Regime A — the agent genuinely finished before the tool.** This is the async-dispatch bug above. The fix is to make the agent span's lifetime encompass the tool's: `await` outstanding tools before `span.end()`. Do **not** "fix" it by reparenting the tool span to the next agent turn or to the trace root — that hides the bug and corrupts the loop reconstruction.
- **Regime B — the tool is genuinely fire-and-forget** (e.g., a background enrichment that outlives the request). In this case the tool should **not** be a child of `invoke_agent` at all; it should be a *link* (`span.add_link(agent_ctx)`) from a span parented to the trace root or to a dedicated `background` span. The GenAI conventions permit links for exactly this decoupling; they do not permit temporal inversion of a parent-child edge.

The boundary is: **if the tool's result is consumed by the agent loop, it must be temporally contained; if it is not consumed, it must not be a child.** There is no valid third state where a consumed tool outlives its consumer.

## Operational Effect

The downstream consequences of getting this wrong are quiet and compounding:

1. **Latency attribution drift.** Backends that compute p99 agent-turn latency as `agent.end - agent.start` will under-report by exactly the tool tail — the portion of tool execution after the agent span closed. In the observed case, p99 was under-reported by 11% for turns that dispatched tools.
2. **Eval-set coverage loss.** Pipelines that filter tool calls to those with positive duration (a common sanity check) silently drop the inverted ones, biasing the eval toward fast tools.
3. **Loop reconstruction errors.** Any consumer that reconstructs the agent loop by sorting child spans by `start_time` under the agent span will place the late tool *outside* the agent, breaking turn boundaries and causing tool-call counts to be attributed to the wrong turn.
4. **Collector-side rejection is not guaranteed.** The OTLP spec does not require a collector to validate temporal containment; assume it will not. The invariant is yours to enforce.

The mitigation is a cheap assertion at span-close time in the agent runtime, plus a trace-level validator in the eval pipeline that flags any `execute_tool` span whose `end_time` exceeds its `invoke_agent` ancestor's `end_time`. The validator is the durable fix; the assertion is the fast one.

```python
from __future__ import annotations

from dataclasses import dataclass


class TemporalContainmentError(RuntimeError):
    """Raised when a GenAI child span outlives its enclosing agent span.

    Under OpenTelemetry GenAI semantic conventions, an `execute_tool` span
    that is a child of an `invoke_agent` span must have start_time >=
    parent.start_time and end_time <= parent.end_time. Structural parentage
    (parent_span_id) is enforced by the SDK; temporal containment is not.
    """


@dataclass(frozen=True)
class SpanInterval:
    span_id: str
    name: str
    parent_span_id: str | None
    start_ns: int
    end_ns: int


def assert_temporal_containment(
    agent: SpanInterval,
    tool: SpanInterval,
) -> None:
    """Validate that `tool` is temporally contained in `agent`.

    Call this before shipping the trace (or in the eval pipeline) to catch
    async-dispatch bugs where the agent span is ended while tool tasks are
    still in flight.
    """
    if tool.parent_span_id != agent.span_id:
        raise TemporalContainmentError(
            f"tool span {tool.span_id!r} is not a child of agent span "
            f"{agent.span_id!r} (parent={tool.parent_span_id!r}); "
            "check that the context was captured at dispatch time"
        )

    if tool.start_ns < agent.start_ns:
        raise TemporalContainmentError(
            f"tool span {tool.span_id!r} starts at {tool.start_ns} before "
            f"agent span {agent.span_id!r} starts at {agent.start_ns}; "
            "tool must be dispatched after the agent span begins"
        )

    if tool.end_ns > agent.end_ns:
        raise TemporalContainmentError(
            f"tool span {tool.span_id!r} ends at {tool.end_ns} after agent "
            f"span {agent.span_id!r} ends at {agent.end_ns} "
            f"(tail={tool.end_ns - agent.end_ns} ns). "
            "Either await outstanding tool tasks before span.end(), or if "
            "the tool is genuinely fire-and-forget, reparent it to the trace "
            "root and add a link to the agent span instead of a parent edge."
        )


def validate_agent_trace(spans: list[SpanInterval]) -> None:
    """Validate all execute_tool spans against their invoke_agent ancestors."""
    by_id = {s.span_id: s for s in spans}
    for span in spans:
        if span.name != "execute_tool":
            continue
        # Walk up to the nearest invoke_agent ancestor.
        cursor = span.parent_span_id
        while cursor is not None:
            ancestor = by_id.get(cursor)
            if ancestor is None:
                raise TemporalContainmentError(
                    f"tool span {span.span_id!r} references missing ancestor "
                    f"{cursor!r}; trace is incomplete"
                )
            if ancestor.name == "invoke_agent":
                assert_temporal_containment(ancestor, span)
                break
            cursor = ancestor.parent_span_id
        else:
            raise TemporalContainmentError(
                f"tool span {span.span_id!r} has no invoke_agent ancestor; "
                "under GenAI conventions execute_tool must be scoped to an "
                "agent invocation or linked to the trace root"
            )
```