# How does an instance detect trace-context divergence when a federated peer silently re-roots spans under a different trace ID **Date of Record:** 2026 November 16 **Scope:** Applies to multi-runtime agent fleets where spans are emitted by two or more independently operated tracing backends and correlated by `trace_id`/`parent_span_id` at a federation gateway. Does not apply to single-runtime deployments, nor to runtimes that share a single W3C `traceparent` authority end-to-end with no gateway translation. ## Core Distinction The failure is not "missing spans." It is *silent trace re-rooting*: a federated peer receives a valid inbound `traceparent` header, fails to honor its `trace_id` field (because of a sampling policy, a local `TraceIdRatioBasedSampler` initialized without the parent context, a stale SDK that regenerates IDs on `start_span`, or a proxy that strips the header), and emits child spans under a *fresh* trace ID. The child spans are internally consistent and pass all local schema validation. The federation gateway, which joins on `trace_id`, therefore reports two complete-looking traces instead of one broken one. Observability dashboards show healthy span counts, low error rates, and zero dropped spans — the only symptom is that the causal chain across the boundary is severed, and it is severed *invisibly*. The critical distinction from ordinary sampling loss: sampled-out spans are *absent*; re-rooted spans are *present but orphaned*. Absence is detectable by count reconciliation. Orphaning is not, because an orphan is a well-formed root from the gateway's point of view. ## What Was Observed In a three-runtime agent pipeline (planner → tool-executor → synthesizer), each runtime emitted spans with correct `parent_span_id` linkage, but the tool-executor's spans carried a `trace_id` that differed from the planner's by a single 16-hex-char suffix regeneration. Root-cause: the executor's runtime was upgraded to an SDK that, when `OTEL_PROPAGATORS` was unset, defaulted to `none` propagation and generated a new trace ID at the first `start_span` call rather than adopting the inbound context. The gateway's join query `WHERE trace_id = ?` returned two disjoint result sets. Detection signal that actually worked: the planner emitted a `span.kind=client` span for the RPC to the executor, and the executor emitted a `span.kind=server` span for the same RPC. Under correct propagation these share a `trace_id` and the server span's `parent_span_id` equals the client span's `span_id`. Under re-rooting, the client span has no matching server span *in the same trace*, and the server span has a `parent_span_id` pointing to a span that exists in *no* trace. That dangling-parent condition — a `parent_span_id` that resolves to nothing anywhere in the federation index — is the invariant violation. ## Condition of Validity The detection rule is only valid when all of the following hold: 1. The gateway indexes spans from *all* federated runtimes before evaluating parent resolution (i.e., a two-phase collect-then-join, not a streaming join that can race). 2. Runtimes are required to emit paired `client`/`server` span kinds at every RPC boundary; if a runtime emits only one side, dangling-parent detection yields false positives. 3. Clock skew between runtimes is bounded below the trace's end-to-end latency, so a server span cannot legitimately precede its client span by more than the skew budget. 4. Batch spans that are *intentionally* root spans (e.g., async job triggers, cron-invoked agents) are excluded via a `span.attributes["federation.root_reason"]` allowlist; otherwise every legitimate root looks like an orphan. If condition 2 fails, fall back to trace-ID continuity across consecutive hops: a hop that changes `trace_id` without a `federation.root_reason` is a violation, but this is weaker (it cannot distinguish a legitimate new root from re-rooting without the allowlist). ## Decision Boundary Do **not** attempt to repair the trace by rewriting the orphan's `trace_id` to match the client span's trace. Rewriting is tempting because it makes dashboards look correct, but it fabricates causal edges that the runtime never asserted: if the executor re-rooted because it *intentionally* started an independent workflow (a fan-out sub-agent with its own lifecycle), rewriting merges two semantically distinct traces and corrupts latency attribution for both. The correct boundary is: **detect and quarantine, never reconcile.** Emit the orphaned subtree to a `federation.quarantine` index keyed by the dangling `parent_span_id`, alert on quarantine rate, and require the owning runtime to fix its propagator configuration. Reconciliation is only permissible when the runtime explicitly declares `federation.correlation_mode=adopt_parent` in its resource attributes, which is a contract, not an inference. ## Operational Effect Once this rule was deployed as a gateway-side pre-join validator, re-rooting incidents went from "discovered weeks later during a latency postmortem" to "paged within one batch interval." The quarantine rate became a leading indicator: a spike correlated with every SDK rollout that touched propagation config, giving the platform team a canary signal independent of error rates. The cost is one extra full-index pass per batch (parent-resolution join), which was measured at ~4% of gateway batch time at the observed span volume. ```python from __future__ import annotations from dataclasses import dataclass, field class TraceRootingDivergence(RuntimeError): """Raised when a federated span's parent cannot be resolved in any runtime.""" @dataclass(frozen=True) class Span: trace_id: str span_id: str parent_span_id: str | None runtime: str kind: str # "client" | "server" | "internal" root_reason: str | None = None # federation.root_reason allowlist @dataclass class FederationIndex: spans: dict[str, Span] = field(default_factory=dict) # span_id -> Span def ingest(self, span: Span) -> None: if span.span_id in self.spans: raise TraceRootingDivergence( f"duplicate span_id {span.span_id} from runtime {span.runtime}" ) self.spans[span.span_id] = span def resolve_parent(self, span: Span) -> Span | None: if span.parent_span_id is None: return None return self.spans.get(span.parent_span_id) def validate_rooting(index: FederationIndex, skew_tolerant_roots: set[str]) -> list[Span]: """Return spans that dangle: their parent resolves to nothing in the federation.""" orphans: list[Span] = [] for span in index.spans.values(): if span.parent_span_id is None: if span.root_reason not in skew_tolerant_roots: orphans.append(span) # bare root with no declared reason continue parent = index.resolve_parent(span) if parent is None: orphans.append(span) continue if parent.trace_id != span.trace_id: # Parent exists but under a different trace: silent re-rooting. raise TraceRootingDivergence( f"span {span.span_id} (runtime {span.runtime}) claims parent " f"{span.parent_span_id} but trace_id diverges: " f"parent={parent.trace_id} child={span.trace_id}" ) return orphans def correlate_pair(client: Span, server: Span) -> None: """Enforce client/server pairing invariant at an RPC boundary.""" if client.kind != "client" or server.kind != "server": raise TraceRootingDivergence( f"correlate_pair requires client/server kinds, got {client.kind}/{server.kind}" ) if client.trace_id != server.trace_id: raise TraceRootingDivergence( f"RPC boundary re-rooted: client trace {client.trace_id} " f"!= server trace {server.trace_id} (runtime {server.runtime})" ) if server.parent_span_id != client.span_id: raise TraceRootingDivergence( f"server span {server.span_id} parent {server.parent_span_id} " f"does not match client span {client.span_id}" ) ```