# What structural signals indicate broken causal linkage between prompt spans, tool spans, and memory operations in agent execution graphs

**Date of Record:** 2026 November 12

**Scope:** Applies to agent runtimes that emit OpenTelemetry-style spans for LLM prompt invocations, tool/function calls, and memory read/write operations, correlated via trace context and parent-child relationships. Does not apply to single-shot RAG pipelines without tool loops, nor to systems that reconstruct causality purely from application-level logs without span linkage.

## Core Distinction

A trace is *structurally valid* when every span has a resolvable parent within the same trace, and *causally valid* when the parent-child edges actually reflect the execution dependency (the child could not have executed before the parent completed or yielded). These diverge in agent runtimes because spans are frequently emitted by three different instrumentation layers — the LLM SDK, the tool executor, and the memory store — each of which may propagate context differently. The non-obvious failure is that a trace can be 100% structurally valid (no orphan spans, no missing parents) while the causal graph is inverted or flattened: tool spans appearing as siblings of the prompt span that triggered them, or memory reads parented to the root span rather than to the retrieval step that requested them. Structural completeness masks causal corruption.

## What Was Observed

Across several agent runtimes, the tell-tale signatures of broken causal linkage were:

1. **Tool spans parented to the root instead of the prompt span.** The tool executor created its own span from an ambient context rather than the span context passed in the tool-call arguments. The graph looked like a star (root → prompt, root → tool) instead of a chain (root → prompt → tool). Duration arithmetic then showed tool time overlapping prompt time, which is physically impossible for a synchronous tool loop.
2. **Memory operations with zero-duration or post-hoc timestamps.** Memory read spans were emitted *after* the prompt span closed, because the memory client batched its telemetry flush. Their `start_time` fell outside the parent's `[start, end]` interval. A child span whose interval is not contained within its parent's interval is a definitive causal-inversion signal.
3. **Duplicate tool spans with divergent parentage.** Retry logic in the executor emitted a second span for the same `tool_call_id` under a different parent (the retry wrapper). Downstream, the graph showed two causally independent tool invocations where there was one logical call, inflating fan-out and breaking idempotency analysis.
4. **Memory writes attributed to the wrong turn.** The memory store keyed spans by session, not by the prompt span that produced the write, so a write triggered by turn N was parented to the root and appeared concurrent with turn N-1's read. The read-after-write dependency that the agent relied on was invisible in the graph.
5. **Broken trace-context propagation across async boundaries.** When the tool executor ran on a separate thread pool, the `contextvars`-based span context was not copied, so tool spans started new traces entirely. These appeared as separate traces rather than orphans — a subtler failure, because per-trace validation passes.

## Condition of Validity

The causal graph is trustworthy only when all of the following hold simultaneously:

- Every non-root span's `[start_time, end_time]` is contained within its parent's interval (with a tolerance for clock skew, typically ≤ a few ms).
- Every `tool_call_id` in a prompt span's completion maps to exactly one tool span, and that tool span's parent is the prompt span whose completion requested it.
- Every memory read/write span is parented to the agent step (prompt or tool span) whose logic issued the operation, and its `start_time` is ≥ that parent's `start_time` and ≤ the parent's `end_time`.
- Span context is propagated across every async/thread boundary; a tool span that begins a new trace while a logical parent span is open is invalid.
- Clock domains are unified or explicitly reconciled; mixing host clocks and monotonic clocks across the three instrumentation layers invalidates containment checks.

If any of these fail, treat the graph as descriptive-only, not causal.

## Decision Boundary

Do **not** attempt to repair the graph by re-parenting spans heuristically at query time. Re-parenting by timestamp proximity produces plausible-but-wrong causality and is worse than an honest flat graph, because downstream analyses (critical-path latency, blame attribution, memory-staleness detection) will silently consume fabricated edges. The correct boundary is: if structural validation passes but containment or `tool_call_id`-to-span bijection fails, **reject the trace as causally invalid and emit a correlation-integrity error** rather than serving it. Repair belongs at the instrumentation layer (context propagation in the executor, span emission before flush, keying memory spans by the issuing prompt span), not at the analysis layer. Serving a causally corrupt trace is acceptable only for raw latency aggregation that ignores parent-child structure.

## Operational Effect

When this boundary is enforced, two things change. First, agent debugging shifts from "the trace looks complete" to "the trace is causally sound," and failures like stale memory reads become visible as broken read-after-write edges rather than as unexplained answer drift. Second, correlation-integrity errors become a first-class SLO: a rising rate of causally invalid traces predicts instrumentation regressions (usually a new async path or a new tool integration) before they manifest as user-visible agent errors. The cost is that some traces are withheld from causal analysis until instrumentation is fixed — an acceptable trade against fabricated causality poisoning the operational record.

```python
from dataclasses import dataclass


class CausalCorrelationError(RuntimeError):
    """Raised when a span graph is structurally complete but causally invalid."""
    pass


@dataclass
class Span:
    span_id: str
    parent_id: str | None
    start_ns: int
    end_ns: int
    kind: str  # "prompt" | "tool" | "memory_read" | "memory_write"
    tool_call_id: str | None = None


CLOCK_SKEW_TOLERANCE_NS = 5_000_000  # 5 ms


def validate_causal_graph(spans: dict[str, Span]) -> None:
    by_parent: dict[str | None, list[Span]] = {}
    for s in spans.values():
        by_parent.setdefault(s.parent_id, []).append(s)

    for s in spans.values():
        if s.parent_id is None:
            continue
        parent = spans.get(s.parent_id)
        if parent is None:
            raise CausalCorrelationError(
                f"span {s.span_id} references missing parent {s.parent_id}"
            )
        if s.start_ns < parent.start_ns - CLOCK_SKEW_TOLERANCE_NS:
            raise CausalCorrelationError(
                f"span {s.span_id} starts before its parent {parent.span_id}: "
                f"causal inversion (child {s.start_ns} < parent {parent.start_ns})"
            )
        if s.end_ns > parent.end_ns + CLOCK_SKEW_TOLERANCE_NS:
            raise CausalCorrelationError(
                f"span {s.span_id} ends after its parent {parent.span_id}: "
                f"interval not contained (child {s.end_ns} > parent {parent.end_ns})"
            )

    # Every tool span must be a child of the prompt span whose completion
    # requested it, keyed by tool_call_id.
    prompt_by_tool_call: dict[str, Span] = {}
    for s in spans.values():
        if s.kind == "prompt" and s.tool_call_id is not None:
            prompt_by_tool_call[s.tool_call_id] = s

    for s in spans.values():
        if s.kind != "tool":
            continue
        if s.tool_call_id is None:
            raise CausalCorrelationError(
                f"tool span {s.span_id} has no tool_call_id; cannot bind to prompt"
            )
        expected_parent = prompt_by_tool_call.get(s.tool_call_id)
        if expected_parent is None:
            raise CausalCorrelationError(
                f"tool span {s.span_id} has tool_call_id {s.tool_call_id} "
                f"with no originating prompt span"
            )
        if s.parent_id != expected_parent.span_id:
            raise CausalCorrelationError(
                f"tool span {s.span_id} parented to {s.parent_id}, "
                f"expected {expected_parent.span_id} (the prompting span)"
            )

    # Memory operations must be parented to the step that issued them,
    # never to the root, and must fall inside their parent's interval.
    for s in spans.values():
        if s.kind not in ("memory_read", "memory_write"):
            continue
        if s.parent_id is None:
            raise CausalCorrelationError(
                f"memory span {s.span_id} parented to root; "
                f"issuing agent step is unrecoverable"
            )
```