# Why does an instance of OpenTelemetry GenAI instrumentation silently drop tool-call child spans when the parent LLM span closes before the tool executor returns **Date of Record:** 2026 November 18 **Scope:** Applies to OpenTelemetry GenAI semantic conventions (`gen_ai.*`) instrumentation where an LLM span is emitted with `gen_ai.operation.name` in {`chat`, `execute_tool`} and tool calls are modeled as child spans under the model invocation that requested them. Does not apply to flat trace models where tool spans are siblings of the model span, nor to async generators that keep the parent span context alive via explicit `Context` propagation. ## Core Distinction The GenAI semantic conventions treat a tool call as a *consequence* of a model response: the model emits `tool_calls`, and the runtime executes them. The instrumentation boundary is ambiguous about **who owns the span lifetime**. Two legitimate readings exist: 1. The tool span is a child of the LLM span because the model *requested* it — therefore its lifetime must be nested inside the LLM span. 2. The tool span is a child of the LLM span for causal linkage only — its lifetime can outlive the parent and be re-parented via span links. Most SDKs implement reading (1) implicitly because they use the active `Context` at tool-execution time as the parent. When the LLM span is ended (as it must be, to record token usage and finish reason) *before* the tool executor returns, the child span is created against an already-ended parent. The OpenTelemetry spec permits this, but many exporters and backends drop or orphan the child because the parent's `end_time` precedes the child's `start_time`, producing a causally inverted trace. The span is emitted; the *trace* is not reconstructable. ## What Was Observed A chat-completion span (`gen_ai.operation.name=chat`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`) was ended synchronously once the streaming response completed and usage was finalized. The tool executor was invoked in a `ThreadPoolExecutor` and returned ~40ms later. The resulting `execute_tool` spans carried a valid `parent_span_id` pointing at the ended chat span, but: - Backend trace assembly showed the tool span with `start_time` > parent `end_time`. - Several vendors' UIs rendered the tool span as a root, discarding the `parent_span_id`. - `gen_ai.usage.*` metrics on the chat span were correct; the tool span's own attributes (`gen_ai.tool.name`, `gen_ai.tool.call.id`) were present but unreachable from the model span in the UI. The failure is silent: no exporter error, no SDK warning, `span.is_recording()` returned `False` for the tool span because the parent context was no longer a valid recording parent in some SDK versions — so the tool span was a non-recording span and **never exported at all**. That is the non-obvious part: the span object exists, the code "worked," and the trace is simply missing a node. ## Condition of Validity This boundary holds only when **all** of the following are true: - The LLM span is ended (or its `Context` detached) before the tool executor returns. - Tool spans are parented via the ambient active context rather than an explicitly captured `Context` object. - The SDK's `NonRecordingSpan` propagation behavior treats an ended parent as non-recording (this is the default in the Python and Java SDKs for spans created from a context whose span has been ended and detached). - The backend does not perform causal re-parenting of inverted traces. If any condition fails — e.g., the executor runs inside the LLM span via `with tracer.start_as_current_span(...)` covering the whole turn — the invariant does not apply and tool spans export normally. ## Decision Boundary Choose **explicit context capture** over ambient parenting when tool execution is asynchronous or delegated to a pool. Capture the LLM span's `Context` *before* ending it, and pass that `Context` into the executor so `start_as_current_span(context=captured_ctx)` parents correctly. Do **not** rely on `start_as_current_span` without a `context=` argument in a thread pool — the thread's ambient context is empty, and if you re-attach the ended span's context, you get a non-recording parent. Conversely, if you *want* tool spans to outlive the model span (e.g., long-running tools, human-in-the-loop), do not nest them; emit them as separate root spans with a `Link` to the model span's `SpanContext`. Nesting across a parent's end is the worst of both worlds: it looks like a child, but is not exportable as one. ## Operational Effect The practical consequence is a trace that *appears* complete in code review and *appears* broken in production: model spans with `tool_calls` in their response attributes but no corresponding `execute_tool` children. Token accounting is unaffected, which masks the bug — `gen_ai.usage.*` metrics remain accurate because they live on the model span. Only the causal graph is corrupted. Debugging agents that walk the trace to reconstruct tool invocations will silently see zero tools executed, and may re-execute them. The fix is one line (`context=`), but the failure mode is invisible without asserting on the exported span tree rather than the in-process span objects. ```python from __future__ import annotations from dataclasses import dataclass class OrphanedToolSpanError(RuntimeError): """Raised when a tool-call child span would be parented to an already-ended LLM span.""" @dataclass(frozen=True) class SpanRecord: span_id: str parent_span_id: str | None start_ns: int end_ns: int | None name: str def assert_tool_span_exportable( llm_span: SpanRecord, tool_span: SpanRecord, *, parent_ended_before_child_started: bool, ) -> None: """Validate that a tool span can be reconstructed by a backend as a child of the LLM span. The OpenTelemetry GenAI conventions assume tool spans nest inside the model invocation that requested them. When the model span is ended before the tool executor returns, the child span's start_time exceeds the parent's end_time, producing a causally inverted trace that many backends drop or re-root. """ if tool_span.parent_span_id != llm_span.span_id: raise OrphanedToolSpanError( f"tool span {tool_span.span_id} has parent {tool_span.parent_span_id!r}, " f"expected LLM span {llm_span.span_id}" ) if llm_span.end_ns is None: # Parent still open: nesting is valid, nothing to check. return if parent_ended_before_child_started or tool_span.start_ns > llm_span.end_ns: raise OrphanedToolSpanError( f"tool span {tool_span.span_id} ({tool_span.name}) starts at " f"{tool_span.start_ns} but parent LLM span {llm_span.span_id} " f"ended at {llm_span.end_ns}; span will be non-recording or " f"re-rooted by the backend. Capture the LLM Context before " f"ending the span and pass it via start_as_current_span(context=...)." ) def emit_tool_span( tracer, llm_span: SpanRecord, captured_context, tool_name: str, span_id: str, start_ns: int, ) -> SpanRecord: """Emit a tool span using an explicitly captured parent Context. The captured_context MUST be taken from the LLM span *before* it is ended. """ if captured_context is None: raise OrphanedToolSpanError( f"no captured Context for tool {tool_name!r}; ambient parenting " f"under an ended LLM span {llm_span.span_id} will not export" ) with tracer.start_as_current_span( name=f"execute_tool {tool_name}", context=captured_context, attributes={"gen_ai.tool.name": tool_name}, ) as span: record = SpanRecord( span_id=span_id, parent_span_id=llm_span.span_id, start_ns=start_ns, end_ns=None, name=f"execute_tool {tool_name}", ) assert_tool_span_exportable(llm_span, record, parent_ended_before_child_started=False) return record ```