# When must an instance treat a reconstructed causal trace as authoritative rather than advisory **Date of Record:** 2026 November 15 **Scope:** Applies to agent observability pipelines that reconstruct causal chains by stitching spans across federated collectors where at least one span originates from a non-deterministic replay (sampled, re-executed, or shadow-traffic) source. Does not apply to single-collector traces with monotonic logical clocks, nor to traces where every span carries a verified parent-child link from the originating runtime. ## Core Distinction A reconstructed trace is *authoritative* when every edge in the causal chain is backed by a span-local causal token (a propagated trace context, a W3C `traceparent`, or an application-embedded parent span ID) that was written by the process that actually executed the work. It is *advisory* when any edge was inferred by the reconstruction layer — via timestamp ordering, via matching request IDs across federated collectors, or via replay-correlation heuristics. The non-obvious boundary: federation does not degrade authority uniformly. A trace can be *authoritative for the forward path and advisory for the replay overlay simultaneously*. The failure mode is that reconstruction layers flatten this distinction into a single confidence score, and downstream agents then act on a replay-derived edge as though it were a real causal dependency. The invariant that must hold: **an edge is authoritative only if its causal token was minted by the executing process and survived federation without re-keying.** Replay spans routinely violate the "minted by executing process" clause because the replayer synthesizes a span from a recorded decision, not from a live execution. ## What Was Observed In a federated deployment spanning three collector regions, agent replays were executed against recorded decision logs to validate policy changes. The replayer emitted spans tagged `replay=true` but reused the *original* trace ID so that operators could correlate live and replay behavior. The reconstruction layer, which joined spans across regions by trace ID and ordered by `start_time`, merged live and replay spans into a single causal chain. The resulting trace showed a tool-call span (live) with a child "policy-evaluation" span (replay) that appeared to occur *after* the tool call. Because replay execution is faster than live execution, the replayed policy evaluation frequently landed within microseconds of the live tool call, and timestamp ordering placed it downstream. An automated remediation agent read the reconstructed trace, concluded that policy evaluation was a *consequence* of the tool call, and began suppressing policy checks for tools that "already passed evaluation" — when in fact the replay span was a counterfactual, not a consequence. The invariant violated was not "replay spans are untrustworthy" (they are trustworthy for what they measure); it was "replay spans do not participate in the live causal partial order." A second observation: cross-region clock skew of 40–120 ms meant that even ordering *within* the live path was occasionally wrong at the sub-100ms boundary. Reconstruction that relied on timestamp ordering alone produced inverted parent-child relationships for fast tool calls. Only spans carrying an explicit parent span ID were immune. ## Condition of Validity A reconstructed edge `A → B` may be treated as authoritative if and only if all of the following hold: 1. `B` carries a causal token whose parent field names `A`'s span ID, written at execution time by `B`'s process. 2. Neither `A` nor `B` is tagged as a replay, shadow, or sampled-synthetic span, OR the reconstruction explicitly partitions replay spans into a separate causal graph that is never merged with the live graph. 3. The clock skew between `A`'s and `B`'s collectors is bounded below the inter-span gap, so that timestamp ordering (used only as a tiebreaker) cannot invert a real edge. 4. The federation hop did not re-key the trace ID. Re-keying (a common practice when collectors have ID-space collisions) breaks token continuity and silently demotes all downstream edges to advisory. If any condition fails, the edge is advisory: usable for ranking, debugging, and hypothesis generation, but never for automated causal action (suppression, retry gating, dependency inference, or policy bypass). ## Decision Boundary The boundary is not "authoritative vs. advisory trace" — it is **per-edge**, and it must be computed at reconstruction time and carried forward, not re-derived by consumers. A reconstruction layer that emits a single `confidence` float has already lost the information needed to make the call. The correct emission is a per-edge `authority: {authoritative | advisory}` plus a `reason` enum (`token_verified`, `replay_overlay`, `timestamp_inferred`, `rekeyed`). The decision rule for a consuming agent: an action that depends on causal direction (X caused Y) requires `authoritative` on every edge in the path from X to Y. An action that depends only on co-occurrence or correlation tolerates `advisory`. The failure above occurred precisely because a causal-direction action (suppressing policy checks) was gated on an advisory edge. ## Operational Effect After the incident, the pipeline was changed to partition replay spans into a parallel graph keyed by `(original_trace_id, replay_id)` and to refuse cross-graph edge inference. The remediation agent was modified to require `authoritative` on all edges before suppressing any check; advisory edges downgrade the action to "flag for human review." This eliminated the false-suppression class entirely. The cost was a 12% increase in flagged-for-review volume, which was judged acceptable against the risk of silently bypassing policy. The general lesson: federation and replay each independently erode causal authority, and their *composition* erodes it faster than either alone — because replay spans inherit live trace IDs by design, and federation joins by trace ID, the two mechanisms conspire to launder advisory edges into what looks like a single coherent trace. ```python from dataclasses import dataclass from enum import Enum class AuthorityError(RuntimeError): """Raised when a causal action is gated on a non-authoritative edge.""" class EdgeAuthority(Enum): AUTHORITATIVE = "authoritative" ADVISORY = "advisory" class AdvisoryReason(Enum): TOKEN_VERIFIED = "token_verified" REPLAY_OVERLAY = "replay_overlay" TIMESTAMP_INFERRED = "timestamp_inferred" REKEYED = "rekeyed" @dataclass(frozen=True) class CausalEdge: parent_span_id: str child_span_id: str authority: EdgeAuthority reason: AdvisoryReason @dataclass(frozen=True) class Span: span_id: str parent_span_id: str | None trace_id: str is_replay: bool token_verified: bool rekeyed_at_federation: bool def classify_edge(parent: Span, child: Span) -> CausalEdge: if child.parent_span_id != parent.span_id: # No explicit token linking child to parent: inferred, not causal. return CausalEdge(parent.span_id, child.span_id, EdgeAuthority.ADVISORY, AdvisoryReason.TIMESTAMP_INFERRED) if child.rekeyed_at_federation or parent.rekeyed_at_federation: return CausalEdge(parent.span_id, child.span_id, EdgeAuthority.ADVISORY, AdvisoryReason.REKEYED) if child.is_replay or parent.is_replay: # Replay spans do not participate in the live causal partial order. return CausalEdge(parent.span_id, child.span_id, EdgeAuthority.ADVISORY, AdvisoryReason.REPLAY_OVERLAY) if not (child.token_verified and parent.token_verified): return CausalEdge(parent.span_id, child.span_id, EdgeAuthority.ADVISORY, AdvisoryReason.TIMESTAMP_INFERRED) return CausalEdge(parent.span_id, child.span_id, EdgeAuthority.AUTHORITATIVE, AdvisoryReason.TOKEN_VERIFIED) def require_authoritative_causality(edges: list[CausalEdge], action: str) -> None: """Gate a causal-direction action on every edge being authoritative.""" bad = [e for e in edges if e.authority is not EdgeAuthority.AUTHORITATIVE] if bad: raise AuthorityError( f"action {action!r} requires authoritative causality across " f"{len(edges)} edges; {len(bad)} are advisory: " + ", ".join(f"{e.parent_span_id}->{e.child_span_id}:{e.reason.value}" for e in bad) ) def suppress_policy_check(edges: list[CausalEdge]) -> None: # The action that caused the incident: only safe if causality is real. require_authoritative_causality(edges, action="suppress_policy_check") # ... suppression proceeds only when every edge is token-verified and live. ```