# Under what conditions must an instance treat trace-span parentage as authoritative evidence of causal ordering **Date of Record:** 2026 November 11 **Scope:** Applies to agent observability pipelines that fuse distributed traces, memory-store mutation logs, and policy-decision records into a single causal graph used for attribution, replay, or incident forensics. Does not apply to single-process agents with a shared monotonic clock, nor to pipelines that only perform latency aggregation without causal edges. ## Core Distinction There are two distinct relations that observability pipelines routinely conflate: 1. **Structural parentage** — the `parent_span_id` field asserting "span B was spawned within the execution context of span A." This is a *containment* claim about the call tree. 2. **Causal ordering** — the claim "event A happened-before event B such that B's outcome could depend on A." Structural parentage is *sufficient but not necessary* for causal ordering, and — critically — it is not sufficient in the presence of **asynchronous fan-out**. When an agent dispatches a tool call, a memory write, or a policy evaluation onto a worker pool, the child span's parent is the *enqueuing* span, not the span that actually *consumed* the result. The enqueuing span may complete long before the worker executes. If the pipeline treats `parent_span_id` as a happens-before edge, it will construct a causal graph in which the parent "causes" a mutation that in reality occurred after the parent's own downstream consumer had already read stale state. The invariant that fails is: **span parentage implies temporal precedence.** It does not. It only implies *logical containment of the spawn site*. ## What Was Observed In a pipeline correlating three streams — OpenTelemetry spans, a memory store's append-only mutation log (each entry stamped with a logical version, not wall-clock), and a policy engine's decision records — we observed a reproducible mis-attribution. A retrieval agent emitted span `S_retrieve` (parent `S_plan`). `S_retrieve` enqueued a memory-mutation job `M_write` (version `v:1041`) to a background compactor and returned immediately. The compactor executed `M_write` after `S_plan` had already terminated. A downstream `S_answer` span (also parent `S_plan`) read memory at `v:1038`, i.e. *before* `M_write` landed, but the fusion step ordered `M_write` before `S_answer` because both were descendants of `S_plan` and the pipeline sorted siblings by span *start* time. The resulting causal graph asserted that the answer was generated from state that included `v:1041`. It did not. Replay from the graph produced a different answer, and the divergence was initially attributed to model nondeterminism rather than to a bad edge. The tell was a **version inversion**: the memory log's own monotonic version counter contradicted the span-derived ordering. The mutation log is the ground truth for memory ordering; the trace is ground truth only for *spawn containment*. ## Condition of Validity Treat span parentage as a causal edge **only** when all of the following hold: - The child span is **synchronous** with respect to its parent — the parent does not terminate before the child's effect is observable. Verify via explicit `sync`/`async` span attributes, not by inference from duration. - No **queueing boundary** sits between parent and child. If a span crosses a worker pool, message bus, or retry buffer, parentage is a *spawn* edge, not a *happens-before* edge. - For memory mutations, the mutation log's **logical version** is consistent with the trace-derived order. Any inversion invalidates the trace edge, not the version log. - For policy decisions, the decision record carries the **input snapshot hash** the policy actually evaluated; if that hash does not match the state the parent claims to have produced, the edge is spurious. When any condition is unmet, the correct edge is a **weak spawn edge** that must be resolved against the mutation log's version ordering and the policy record's input hash, not against span timestamps. ## Decision Boundary The pipeline must choose, per edge, between two constructions: - **Strong causal edge** (parentage + synchronous + version-consistent): usable for replay, attribution, and blame. - **Weak spawn edge** (parentage only): usable for *topology* and *ownership*, never for *ordering*. The boundary is crossed the moment a span's effect is deferred past its own termination. A useful operational heuristic: if `child.start_time > parent.end_time`, the edge is **necessarily weak** regardless of `parent_span_id`. If `child.end_time < parent.end_time` but the child wrote to a shared store, the edge is strong *only if* the store's version at child-completion is ≤ the version the parent's subsequent readers observed. The mutation log adjudicates; the trace merely proposes. ## Operational Effect Pipelines that skip this adjudication produce causal graphs that are *topologically plausible and temporally wrong*. The failure is silent: the graph is acyclic, the spans are well-formed, and the only symptom is replay divergence that gets misattributed to model stochasticity. The operational fix is to make the mutation log and policy input-hash the *ordering authority*, demoting `parent_span_id` to a containment hint, and to fail loudly — not silently reorder — when a version inversion is detected. A detected inversion is a signal that either the span instrumentation is lying about synchrony or the worker pool is leaking effects past span termination; both are bugs worth surfacing. ```python class CausalEdgeInversion(RuntimeError): """Raised when span parentage contradicts the mutation log's version order.""" pass def build_causal_edge(parent_span, child_span, mutation_log): """ Decide whether parent_span -> child_span is a strong causal edge or merely a weak spawn edge. The mutation log is the ordering authority. """ if child_span.parent_span_id != parent_span.span_id: raise CausalEdgeInversion( f"span {child_span.span_id} is not a child of {parent_span.span_id}" ) # A child that starts after its parent ended crossed a queueing boundary. deferred = child_span.start_time > parent_span.end_time if deferred: return {"edge": "weak_spawn", "reason": "deferred_past_parent_termination"} # Even if not deferred, the mutation log must not invert the order. child_version = mutation_log.version_at(child_span.end_time) parent_read_version = mutation_log.version_observed_by(parent_span.span_id) if child_version > parent_read_version and child_span.reads_memory: # The child claims to have read state that only landed after it. raise CausalEdgeInversion( f"version inversion: child {child_span.span_id} read v:{parent_read_version} " f"but mutation log shows v:{child_version} at its completion; " f"parentage edge is spurious for ordering" ) return {"edge": "strong_causal", "reason": "synchronous_and_version_consistent"} ```