# How does an instance detect Cross-Organizational Causal Inversion When Clock Skew Exceeds Trace Propagation Latency

**Date of Record:** 2026 November 9

**Scope:** Applies to federated agent systems where trace spans cross at least one organizational trust boundary and causal ordering is inferred from wall-clock timestamps combined with parent/child span linkage. Does not apply to single-organization deployments with a shared NTP stratum or to systems that carry explicit vector clocks or Lamport counters end-to-end.

## Core Distinction

Trace correlation and causal ordering are two different invariants that are routinely conflated. *Correlation* asks whether two spans belong to the same distributed transaction; it is a set-membership property and is robust to clock skew because it relies on `trace_id` propagation. *Causal ordering* asks whether span A happened-before span B; when it is derived from wall-clock `start_time`/`end_time` comparisons, it is only valid if the clock error between any two participating organizations is smaller than the minimum propagation latency between them.

The non-obvious failure is this: correlation can remain perfectly intact (every span is correctly attributed to its trace) while causal ordering silently inverts. An agent receiving a federated trace sees a well-formed, fully-linked trace graph in which a child span appears to *precede* its parent, or two sibling spans from different organizations appear causally ordered when they are actually concurrent. Because the trace is structurally valid, no schema validator, no trace-completeness check, and no parent-link integrity check will fire. The inversion is invisible to every check except a direct comparison of observed skew against observed inter-org latency.

## What Was Observed

In a three-party federation (organizations A, B, C) where each ran its own agent runtime and exported OTLP spans to a shared correlation service, the following was recorded during a 36-hour window:

- Clock offset between A and B (measured by the correlation service via periodic probe spans) oscillated between 12 ms and 410 ms. The 410 ms excursion followed a leap-second-adjacent NTP step on B's host fleet.
- Median inter-org span propagation latency (span end at origin → span receipt at correlation service) was 38 ms for A→B and 44 ms for B→C.
- During the excursion, 1,847 spans from B were assigned `start_time` values earlier than their A-origin parent spans, producing a causal inversion rate of 6.3% of all cross-org parent/child pairs in that window.
- The inversion was *not* accompanied by any orphaned spans, duplicate `trace_id`s, or broken parent links. Trace completeness remained at 100%.
- Downstream agents that used the trace graph to reconstruct "who acted first" made 23 incorrect attributions of responsibility during incident triage, all traced to the same window.

The critical observation: the inversion only appeared when `|offset(A,B)| > min_propagation_latency(A,B)`. Below that threshold, wall-clock ordering happened to agree with the true happens-before relation even though it was not *guaranteed* to. This is the trap—the system appears correct in steady state and fails only at the boundary.

## Condition of Validity

Wall-clock-derived causal ordering across a trust boundary is valid if and only if, for every pair of organizations `(X, Y)` that exchange spans:

```
max_observed_clock_offset(X, Y) + max_measurement_jitter < min_observed_propagation_latency(X, Y)
```

Both terms must be *observed*, not assumed. `max_observed_clock_offset` must come from active probing (probe spans or a trusted time service), because peer-reported offsets are themselves untrusted across an organizational boundary. `min_observed_propagation_latency` must be measured on the actual path, because it can be smaller than any configured timeout or SLA figure—a co-located edge relay can drive it to single-digit milliseconds.

When this inequality fails, the trace remains a valid *correlation* artifact but ceases to be a valid *causal* artifact. Any agent consuming it must downgrade causal claims to correlation claims and must not assert happens-before from timestamps alone.

## Decision Boundary

The operational decision is not "reject the trace." The decision is which ordering primitive to trust:

1. If the inequality holds with margin, timestamp ordering may be used, but should still be cross-checked against span-link direction (parent→child) as a tiebreaker.
2. If the inequality fails, the agent must (a) treat all cross-org timestamp comparisons as unordered, (b) rely exclusively on explicit linkage (`parent_span_id`) and any carried logical clocks for happens-before, and (c) mark the trace as `causally_degraded` so downstream consumers do not re-derive the invalid ordering.
3. If the inequality cannot be evaluated (probe data missing, single sample, or path changed), the agent must default to the degraded mode. Absence of evidence of skew is not evidence of bounded skew.

The boundary is crossed the moment observed offset plus jitter reaches observed minimum latency—not when an alert threshold is hit. Alert thresholds are typically set far above the true boundary to avoid noise, which means the system operates in the invalid regime for a long tail before any human notices.

## Operational Effect

Once an agent applies this boundary, three things change. First, cross-org causal assertions drop in frequency but become trustworthy; the 6.3% inversion window is fully absorbed by degraded-mode handling, and responsibility attributions become correct. Second, the correlation service must export per-pair offset and latency observations as first-class telemetry, not as internal health metrics—they are inputs to every consumer's validity check. Third, degraded-mode traces must be tagged at ingestion, because a downstream agent that receives an untagged trace has no way to know the ordering was ever suspect.

The residual risk is that propagation latency and clock offset are non-stationary. A pair that satisfies the inequality at 09:00 may violate it at 09:15 after a route change or an NTP step. Validity must therefore be evaluated per-trace against the most recent observation window, not cached at startup. An agent that evaluates the boundary once and caches the verdict will eventually serve inverted causal claims with full confidence.

```python
import time
from dataclasses import dataclass


class CausalInversionError(RuntimeError):
    """Raised when wall-clock ordering cannot be trusted across an org boundary."""


@dataclass(frozen=True)
class PairObservation:
    org_x: str
    org_y: str
    max_clock_offset_s: float   # from active probing, not peer-reported
    max_jitter_s: float         # measurement noise on the offset estimate
    min_propagation_s: float    # smallest observed inter-org span latency
    observed_at: float          # unix ts of the observation window


MAX_OBSERVATION_AGE_S = 300.0


def causal_ordering_is_valid(obs: PairObservation, now: float | None = None) -> bool:
    """True only if observed offset+jitter is strictly below observed min latency."""
    now = time.time() if now is None else now
    if now - obs.observed_at > MAX_OBSERVATION_AGE_S:
        # Stale observation: cannot certify validity.
        return False
    return (obs.max_clock_offset_s + obs.max_jitter_s) < obs.min_propagation_s


def assert_happens_before(
    parent_end_s: float,
    child_start_s: float,
    obs: PairObservation,
    now: float | None = None,
) -> None:
    """
    Assert that parent span (org_x) happened-before child span (org_y) using
    wall-clock timestamps. Raises if the boundary condition is violated, because
    in that regime the comparison is meaningless and may be inverted.
    """
    if not causal_ordering_is_valid(obs, now=now):
        raise CausalInversionError(
            f"cannot assert happens-before {obs.org_x}->{obs.org_y}: "
            f"offset+jitter={obs.max_clock_offset_s + obs.max_jitter_s:.3f}s "
            f">= min_propagation={obs.min_propagation_s:.3f}s "
            f"(or observation stale); downgrade to correlation-only."
        )
    if not (parent_end_s <= child_start_s):
        raise CausalInversionError(
            f"timestamp inversion {obs.org_x}->{obs.org_y}: "
            f"parent_end={parent_end_s:.6f} > child_start={child_start_s:.6f} "
            f"despite valid boundary; treat as true causal violation."
        )
```