# Why does an instance of "clean" trace context silently sever the audit chain during agent incident replay? **Date of Record:** 2026 November 17 **Scope:** Applies to multi-hop agent systems where safety evaluations, taint-propagation telemetry, and incident replay all depend on a shared trace/span correlation identifier that is propagated through tool calls, sub-agent invocations, and model-to-model handoffs. Does not apply to single-process agents with no external tool boundary, nor to systems where each hop mints a fresh root trace and correlation is reconstructed offline by content hashing. ## Core Distinction There are two distinct meanings of "trace continuity" that operational dashboards routinely conflate: 1. **Transport continuity** — the `trace_id`/`span_id` pair survives serialization across every process boundary, so the collector can stitch a single flame graph. 2. **Semantic continuity** — the *provenance* of the data flowing through the agent (untrusted user input, retrieved document text, tool-returned payloads, prior model output) remains attributable to its origin at every downstream decision point. An agent trace can be perfectly continuous in the transport sense while being semantically severed. The failure is non-obvious because the observability UI shows one connected trace, the safety eval binds to that trace, and the replay tool reconstructs the run — yet the taint labels that should gate a downstream action were dropped at a boundary that emitted a *new, valid* span. The audit chain looks intact and is not. ## What Was Observed During replay of an incident in which an agent executed a destructive tool call on content that originated from an untrusted retrieved document, the following held simultaneously: - The trace was a single connected tree. `trace_id` was identical across the orchestrator, the retrieval tool, the summarizer sub-agent, and the executor. - The taint-propagation telemetry recorded the document span as `taint:untrusted` with a correct origin pointer. - The safety eval that should have blocked the executor ran against the trace, found the trace linked, and returned `pass` — because it keyed on `trace_id`, and the untrusted span and the executor span shared it. - The executor span carried `taint:clean`. The seam was a **sub-agent invocation that re-rooted its local context**. The sub-agent framework constructed a fresh `SpanContext` for its internal reasoning (to isolate its own retries), inherited the parent `trace_id` for transport purposes, but did not inherit the `taint` baggage item. Downstream, the executor saw a parent span with no taint attribute, applied its default (`clean`), and proceeded. The taint did not "leak" — it was *dropped at a span that was structurally valid*, so no invariant monitor fired. The critical observation: **taint propagation is not a property of the trace graph; it is a property of the baggage channel, and baggage is silently re-initialized by any component that constructs a new context rather than deriving one.** Trace-linking and taint-linking use different propagation mechanisms (context propagation vs. baggage propagation) and can diverge without any transport error. ## Condition of Validity The failure is reachable only when all of the following hold: - The agent topology includes at least one **context-constructing boundary** (sub-agent spawn, tool sandbox, model handoff, or async task) that emits a new span context instead of extending the caller's. - The taint label is carried in **baggage** (or an equivalent side-channel) rather than encoded as a mandatory field on the span attributes that the boundary is contractually required to copy. - The downstream consumer's taint resolution uses a **default-on-absence** policy (`clean` when no label is present) rather than **default-on-absence = quarantine/unknown**. - The safety eval's linkage check keys on `trace_id` alone, so transport continuity masks semantic discontinuity. If any one is false, the failure does not occur: a boundary that extends context preserves baggage; a mandatory-copy contract fails loudly on omission; a fail-closed default quarantines instead of executing; a taint-aware eval detects the missing label. The dangerous configuration is the *combination* — each individual choice is locally reasonable and defensible. ## Decision Boundary The operational rule is a single, sharp line: > **A span that shares a `trace_id` but does not carry an explicit, inherited taint label MUST NOT be treated as clean. It must be treated as `unknown`, and `unknown` must be non-executable for any action classified as irreversible.** This inverts the common default. The tempting default — "no taint label means clean, because clean is the common case" — is exactly what converts a dropped-baggage bug into a silent safety bypass. The decision boundary is therefore not "is the trace connected?" (it will be) but "was the taint label *explicitly present on this span and its ancestors since the last known origin?*" Absence of evidence of taint is not evidence of cleanliness when the propagation mechanism can silently reset. Concretely: the executor must verify that the taint channel is *intact* from the origin span to itself, not merely that a trace path exists. The corollary for replay: an incident replay is only valid if it reproduces the taint channel state at each boundary, not just the span topology. A replay that reconstructs the graph but re-derives taint from defaults will "reproduce" the incident as a clean run and misattribute causality. ## Operational Effect When this boundary is respected, three things change: - **Safety evals become taint-aware, not trace-aware.** The eval asserts a propagation invariant (`taint label present on every span between origin and sink`) rather than a connectivity invariant. This catches the re-rooting seam that connectivity checks miss. - **Incident replay becomes causally faithful.** Replay tooling must snapshot and restore the baggage/taint channel at every context-constructing boundary; if it cannot, it must refuse to certify the replay as reproducing the incident, rather than emit a misleading "clean" reconstruction. - **The blast radius is bounded by the irreversible-action gate.** Because `unknown` is non-executable for irreversible actions, a dropped label degrades to a blocked action and a quarantine alert — a visible, recoverable failure — instead of a silent destructive execution. The residual cost is false positives: legitimate re-rooted spans with no taint will be quarantined until the boundary is fixed to extend context or explicitly declare `taint:clean` with a provenance pointer. That cost is the price of making the seam *loud*. The alternative — trusting trace continuity as a proxy for taint continuity — is precisely the non-obvious invariant whose violation this record documents. ```python # Operational guard for the trace/taint continuity seam. # Enforces: transport continuity does NOT imply semantic (taint) continuity. from dataclasses import dataclass, field from typing import Optional class TaintContinuityError(RuntimeError): """Raised when a span shares a trace but lacks an inherited taint label, i.e. the semantic audit chain was severed at a context-constructing boundary.""" pass @dataclass class Span: trace_id: str span_id: str parent_id: Optional[str] taint: Optional[str] = None # None == label was dropped, NOT "clean" origin_span_id: Optional[str] = None # set only on the span where taint originated IRREVERSIBLE_ACTIONS = {"delete", "transfer_funds", "send_email_external", "execute_shell"} def assert_taint_continuity(span: Span, ancestors: dict[str, Span]) -> str: """Walk parent chain from `span` back to its taint origin. Returns the resolved taint. Raises if the channel was severed.""" if span.taint is None: # A span with no label is UNKNOWN, never implicitly clean. raise TaintContinuityError( f"span {span.span_id} in trace {span.trace_id} has no taint label; " f"treating as 'unknown' and refusing to resolve to 'clean'" ) if span.origin_span_id is not None: return span.taint # Must reach a labelled ancestor with no gap. cur = span while cur.parent_id is not None: parent = ancestors.get(cur.parent_id) if parent is None: raise TaintContinuityError( f"trace {span.trace_id}: parent {cur.parent_id} of {cur.span_id} " f"missing from context; cannot prove taint continuity" ) if parent.taint is None: raise TaintContinuityError( f"trace {span.trace_id}: taint channel severed at span {parent.span_id} " f"(re-rooted context dropped baggage); downstream taint unprovable" ) if parent.origin_span_id is not None: return parent.taint cur = parent raise TaintContinuityError( f"trace {span.trace_id}: no taint origin reachable from {span.span_id}" ) def gate_action(span: Span, ancestors: dict[str, Span], action: str) -> str: """Gate an action on resolved taint. Unknown taint blocks irreversible actions.""" try: taint = assert_taint_continuity(span, ancestors) except TaintContinuityError: if action in IRREVERSIBLE_ACTIONS: raise # fail closed: quarantine, do not execute return "quarantine" # reversible action: degrade to quarantine, still visible if taint == "untrusted" and action in IRREVERSIBLE_ACTIONS: raise TaintContinuityError( f"action '{action}' on span {span.span_id} blocked: taint=untrusted" ) return "allow" ```