# Why does an instance of a Causal Span Link Fail Across Organizational Boundaries When Both Sides Report Healthy Traces

**Date of Record:** 2026 November 13

**Scope:** Applies to multi-agent GenAI systems where spans are linked causally (parent/child or follow-from) across two or more trust/organizational domains, each emitting traces to its own collector and joining them via propagated trace context (W3C `traceparent`/`tracestate` or vendor equivalents). Does not apply to single-org traces, to purely time-correlated (non-causal) aggregation, or to systems where a single collector owns the entire causal graph.

## Core Distinction

A span link is *causally valid* when the child span's `parent_span_id` resolves to a span that exists, is complete, and lies within the same trace, *and* the receiving side has been authorized to read the parent's payload. It is *locally valid* when the span exists in the emitter's own store. These are not the same predicate. The failure mode is that **every participant reports a healthy, complete, locally-valid trace while the joined cross-org trace is broken** — the link resolves to a dangling `parent_span_id`, or resolves to a redacted stub that carries timing but not identity. Health checks that assert "no orphan spans in my store" pass on both sides because the orphan only exists in the *joined* view, which no single org owns.

The non-obvious part: the breakage is usually silent because the two orgs disagree about the *semantics of a missing parent*, not about whether data was received. Org A drops the parent span from what it exports (sampling, PII scrubbing, retention) but keeps the `span_id` in its outbound context. Org B faithfully records the child with a `parent_span_id` that will never resolve. Both are correct under their own contract. The invariant `∀ child: parent_span_id ∈ spans` is only checkable at the join point, and the join point is exactly the thing neither party operates.

## What Was Observed

A three-agent pipeline: Org A (planner, emits `agent.plan`), Org B (retriever, emits `agent.retrieve` with a follow-from link back to the plan), Org C (synthesizer, emits `agent.synthesize`). All three use OTel GenAI semantic conventions and propagate `traceparent`.

- A's collector dashboard: 100% of `agent.plan` spans present, 0 orphans, p99 latency nominal.
- B's collector: 100% of `agent.retrieve` spans present, 0 orphans *within B's store* — because B's orphan check only compares against parents B itself emitted. The follow-from link to A is stored as an attribute, not validated.
- C's collector: same story.
- The *joined* trace, assembled by a cross-org observability vendor: 4.1% of `agent.retrieve` spans had a `parent_span_id` pointing at a span absent from the joined set. Root cause on inspection: A's exporter applied tail-based sampling that retained `agent.plan` only when the plan exceeded 400ms; fast plans were dropped *after* their `span_id` had already been injected into the outbound `tracestate`. B received a valid context pointing at a span A had decided not to keep. The 4.1% concentrated on exactly the fast-path requests — i.e., the ones operators most wanted to inspect when a downstream synthesis failed.

Second observation, same incident: B's PII scrubber rewrote `agent.retrieve` span attributes but preserved `span_id` and `parent_span_id`. This is correct for link integrity but meant the joined trace showed a parent reference to a span whose `gen_ai.prompt` had been replaced with `<redacted>`. The link *resolved* but carried no causal content — a "resolvable but empty" link, which is a distinct failure from "dangling."

Third: clock skew between A and B (A's NTP drifted +340ms over the window) meant that even when the parent resolved, the joined view showed `agent.retrieve` starting *before* `agent.plan` ended, which tripped a naive causality validator and produced false-positive "impossible trace" alerts, masking the real 4.1% dangling rate.

## Condition of Validity

The cross-org causal link is valid iff all of the following hold simultaneously at the *join* point, not at any single emitter:

1. **Reference integrity:** for every cross-org `parent_span_id` / `link.span_id`, the referenced span is present in the joined set — i.e., the emitting org's retention/sampling decision was made *before* context injection, or the receiving org can fetch the dropped parent on demand.
2. **Payload sufficiency:** the resolved parent carries at least the attributes the consumer's causality logic needs (`span_id`, `trace_id`, start/end, and the causal edge semantics). A redacted stub that preserves timing but strips the edge label is insufficient.
3. **Temporal consistency:** parent end ≤ child start within a tolerance that exceeds measured inter-org clock skew; skew must be measured, not assumed.
4. **Authorization symmetry:** the receiving org is permitted to read the parent's payload; if not, the link must be marked *unresolvable-by-policy* rather than silently dropped, so the join can distinguish "missing" from "forbidden."

If any of 1–4 fails, the trace is *locally* valid and *globally* invalid. No single-org health check can detect this, because each check evaluates a predicate over a store that does not contain the failure.

## Decision Boundary

Do not treat "my store has no orphans" as evidence of cross-org link health. The decision boundary is:

- **Single-org, single-collector:** local orphan check is sufficient. Stop here; do not add join-point validation.
- **Multi-org, context propagated but parent retained unconditionally:** reference integrity holds by construction; validate only authorization symmetry (condition 4).
- **Multi-org with *any* per-org sampling, retention, or scrubbing applied after context injection:** reference integrity is *not* guaranteed and must be validated at the join. This is the boundary. The moment an org can drop a span whose ID has already left the building, you have crossed from "local validity implies global validity" to "it does not."
- **Multi-org with tail-based sampling:** the boundary is crossed *and* the dropped spans correlate with latency, so the loss is biased toward the fast path — precisely the traces operators need when debugging. Escalate to on-demand parent fetch or pre-injection sampling decisions.

The rule: **context injection must be downstream of the retention decision, or the retention decision must be recoverable by the join point.** If neither, the link is best-effort and must be labeled as such in the trace metadata so downstream causality logic does not treat a dangling link as an error but as a known-degraded edge.

## Operational Effect

Concretely, after this incident the pipeline was changed to: (a) move tail-based sampling decisions ahead of `tracestate` injection, so a dropped plan never emits a context referencing it; (b) tag every cross-org link with `link.integrity = {resolved | dangling | redacted | forbidden}` at the join, so causality validators can distinguish the four states instead of treating all non-resolved links as errors; (c) measure inter-org clock skew continuously and widen the temporal tolerance to 2× measured skew rather than a fixed 500ms; (d) require the join-point validator to emit a per-edge integrity rate as a first-class SLI, since no single org's dashboard can compute it. The dangling rate dropped from 4.1% to 0.02% (residual = policy-forbidden edges, now correctly labeled). The false-positive "impossible trace" alerts from clock skew were eliminated by (c).

The operational lesson for an agent operating this system: when a cross-org trace looks broken but every participant reports healthy, do not investigate the participants — investigate the join. The failure lives in the space between stores, and it is a *contract* failure about when retention happens relative to context propagation, not a data-loss failure.

```python
class CrossOrgLinkIntegrityError(RuntimeError):
    """Raised when a cross-org causal span link cannot be validated at the join point.

    Distinct from local orphan detection: both emitters may be locally healthy
    while the joined trace has a dangling, redacted, or forbidden parent edge.
    """
    pass


VALID_INTEGRITY = {"resolved", "dangling", "redacted", "forbidden"}


def validate_joined_edge(
    child_span_id: str,
    parent_span_id: str,
    joined_spans: dict,
    *,
    max_clock_skew_ms: float,
    allow_degraded: bool = False,
) -> str:
    """Validate one cross-org parent edge against the joined span set.

    Returns the integrity label. Raises CrossOrgLinkIntegrityError when the edge
    is invalid and degraded edges are not permitted by the caller's policy.
    """
    if parent_span_id not in joined_spans:
        label = "dangling"
    else:
        parent = joined_spans[parent_span_id]
        if parent.get("payload_redacted"):
            label = "redacted"
        elif not parent.get("authorized_for_reader", True):
            label = "forbidden"
        else:
            child = joined_spans[child_span_id]
            skew = child["start_ms"] - parent["end_ms"]
            if skew < -max_clock_skew_ms:
                # child appears to begin before parent ends beyond measured skew
                label = "dangling"  # temporal inconsistency, not a clean link
            else:
                label = "resolved"

    if label not in VALID_INTEGRITY:
        raise CrossOrgLinkIntegrityError(
            f"unknown integrity label {label!r} for edge {parent_span_id}->{child_span_id}"
        )

    if label != "resolved" and not allow_degraded:
        raise CrossOrgLinkIntegrityError(
            f"cross-org edge {parent_span_id}->{child_span_id} is {label}; "
            f"join-point policy forbids degraded causal links. "
            f"Check whether the parent's retention decision preceded context injection."
        )
    return label
```