# What structural signals indicate causal span detachment between LLM reasoning and tool execution in agent observability pipelines **Date of Record:** 2026 November 14 **Scope:** Applies to agent observability pipelines that emit LLM inference spans, tool-execution spans, and memory read/write spans as a single causally-linked trace, where spans are correlated by trace/span IDs and parent-child references. Does not apply to single-process synchronous agents that emit a single flat span, nor to batch evaluation harnesses that do not preserve intra-request causality. ## Core Distinction The failure mode is not "missing spans" and not "orphaned spans." It is **causal detachment**: the tool span and the LLM span both exist, both carry the correct `trace_id`, both have plausible timestamps, and yet the parent-child edge between them is either absent, inverted, or attached to the wrong LLM turn. The pipeline reports a complete, healthy trace with zero dropped spans, while the actual causal chain — "this reasoning step decided to invoke this tool with these arguments" — has been silently severed or misattributed. The non-obvious invariant: **span completeness does not imply causal integrity.** A trace can be 100% complete by count and 0% correct by edge structure. Standard observability checks (span count, error rate, latency percentiles, orphan detection) all pass, because detachment produces well-formed spans with well-formed IDs. The only signals that reveal it are *structural*: edge topology, temporal containment, and argument-provenance consistency. ## What Was Observed Across three production agent deployments (ReAct-style tool loops, 40–200 tool calls per session, memory subsystem using a vector store with synchronous read/write), the following detachment signatures recurred: 1. **Parallel tool-call fan-out collapse.** When the LLM emits N tool calls in one turn, the runtime frequently spawns N goroutines/tasks that each emit their tool span with `parent_span_id` set to the *LLM span that started the turn*. This is correct. But under concurrency >~8, the tracing SDK's context propagation (async-local / contextvar / threadlocal) is not always inherited by the worker, and the tool span is emitted with `parent_span_id = NULL` while retaining the correct `trace_id`. The span is not orphaned by `trace_id` lookup, so orphan detectors pass it. The edge is gone. 2. **Memory-operation re-parenting.** Memory read spans (vector lookup) and write spans (upsert) are emitted by the memory subsystem, which is often a shared client instantiated *before* the agent turn. Its tracing context is captured at client-construction time, not call time. Result: every memory span in a session attaches to the *first* LLM span of the session, regardless of which turn actually triggered the read. The trace shows a memory read on turn 1 that was logically caused by turn 14. Structurally plausible, causally false. 3. **Retry shadowing.** On LLM timeout/retry, the retried LLM span is emitted with a new `span_id` but the tool calls it subsequently dispatches were already bound to the *first* attempt's span ID via a captured context. The tool spans attach to a span that the backend ultimately marks `ERROR`/abandoned, so the successful tool execution appears to be a child of a failed reasoning step. 4. **Timestamp inversion under async flush.** Tool spans are flushed asynchronously; LLM spans synchronously. Under load, the tool span's `start_time` can precede its parent LLM span's `end_time` by tens of milliseconds even when the tool was invoked strictly after the LLM returned. This is benign in isolation but becomes a detachment signal when combined with (1) or (3): the pipeline's temporal validator sees "tool started before reasoning finished" and either drops the edge or re-parents to the previous turn. ## Condition of Validity The detachment is *undetectable* by count-based and error-based monitors and *detectable* only when all of the following hold: - The pipeline preserves raw `parent_span_id` (does not reconstruct edges from timestamps at query time). If the backend reconstructs parentage from `(trace_id, start_time)` ordering, detachment is masked as a sort artifact and cannot be attributed. - Context propagation is async-aware. If the runtime uses a synchronous context (e.g., explicit `context` parameter threading), detachment cannot occur — but neither can the concurrency that makes it worth detecting. - The memory client is not re-instantiated per turn. Per-turn client construction eliminates signature (2) but is itself a latency regression; most deployments accept the shared client and inherit the bug. When any of these conditions is violated, the structural signals below are either unavailable or produce false positives, and the operational guidance does not apply. ## Decision Boundary Distinguish **benign structural noise** from **causal detachment** using three independent checks, all of which must agree before flagging: - **Edge-completeness check:** For every tool span in a trace, `parent_span_id` must resolve to an LLM span within the same `trace_id`, and that LLM span's `end_time` must be ≤ the tool span's `start_time` (allowing a configured clock-skew slack, typically 50–150 ms for async flush). A tool span with `parent_span_id = NULL` inside a trace that has ≥1 LLM span is detached, not orphaned. - **Argument-provenance check:** The tool span's input arguments must appear (substring or structured match) in the parent LLM span's recorded output/tool-call payload. If the parent LLM span's payload does not contain the arguments the tool executed with, the edge is misattributed — the tool was caused by a *different* LLM turn. This is the decisive signal for signature (2) and (3). - **Turn-ordinal check:** For memory spans, the parent LLM span must be the *most recent* LLM span preceding the memory span's `start_time` within the trace. If a memory span's parent is earlier than the immediately-preceding LLM span, the memory client captured a stale context. The boundary: flag as detachment when edge-completeness fails **or** argument-provenance fails **or** turn-ordinal fails. Do not flag on timestamp inversion alone — it is a known async-flush artifact and re-parenting on it corrupts otherwise-correct traces. Do not flag on `parent_span_id = NULL` when the trace has zero LLM spans — that is a legitimate top-level tool invocation (e.g., a health check). ## Operational Effect Detached traces are not merely cosmetic. Downstream consumers of the trace graph — cost attribution, failure root-cause analysis, and any automatic prompt/tool optimization loop — operate on the edge structure. A detached tool span under-counts the LLM turn's tool cost; a re-parented memory span attributes retrieval latency to the wrong reasoning step; a retry-shadowed tool span teaches the optimizer that a failed reasoning path produced a successful tool result. The compounding effect is that the agent's self-improvement loop learns a causal model that is structurally valid and semantically wrong — and because the pipeline reports zero dropped spans, no alert fires. The practical mitigation is to emit the parent-child edge from the *dispatch site* (where the tool is actually invoked), not from the span-emission site (where the tracing SDK happens to run), and to validate argument-provenance at ingest rather than at query time. The check below implements the three-signal boundary and fails closed on any trace that cannot prove causal integrity. ```python from dataclasses import dataclass from typing import Optional class CausalDetachmentError(RuntimeError): """Raised when a tool or memory span cannot be proven causally attached to the LLM span that actually caused it.""" @dataclass class Span: span_id: str parent_span_id: Optional[str] trace_id: str kind: str # "llm" | "tool" | "memory_read" | "memory_write" start_time: float # monotonic seconds end_time: float payload: str # recorded output / tool-call args / memory query CLOCK_SKEW_SLACK = 0.150 # 150 ms for async flush def _llm_spans(trace: list[Span]) -> list[Span]: return sorted((s for s in trace if s.kind == "llm"), key=lambda s: s.start_time) def _most_recent_preceding_llm(trace: list[Span], t: float) -> Optional[Span]: prior = [s for s in _llm_spans(trace) if s.end_time <= t + CLOCK_SKEW_SLACK] return prior[-1] if prior else None def validate_causal_integrity(trace: list[Span]) -> None: by_id = {s.span_id: s for s in trace} llms = _llm_spans(trace) if not llms: return # no reasoning spans; top-level tool invocation is legitimate for span in trace: if span.kind not in ("tool", "memory_read", "memory_write"): continue # Signal 1: edge completeness. parent = by_id.get(span.parent_span_id) if span.parent_span_id else None if parent is None or parent.kind != "llm": raise CausalDetachmentError( f"span {span.span_id} ({span.kind}) has no resolvable LLM parent " f"(parent_span_id={span.parent_span_id!r}); detached edge inside " f"trace {span.trace_id}" ) # Signal 2: argument provenance. if span.kind == "tool" and span.payload not in parent.payload: raise CausalDetachmentError( f"span {span.span_id} executed with payload {span.payload!r} not " f"present in parent LLM span {parent.span_id} payload; edge is " f"misattributed to a different reasoning turn" ) # Signal 3: turn ordinal for memory operations. if span.kind in ("memory_read", "memory_write"): recent = _most_recent_preceding_llm(trace, span.start_time) if recent is not None and recent.span_id != parent.span_id: raise CausalDetachmentError( f"memory span {span.span_id} parented to {parent.span_id} but " f"most recent preceding LLM span is {recent.span_id}; memory " f"client captured a stale tracing context" ) ```