# What structural signals indicate causal trace corruption at agent handoff boundaries

**Date of Record:** 2026 November 8

**Scope:** Applies to multi-agent systems that propagate W3C `traceparent`/`tracestate` and OpenTelemetry span context across asynchronous agent handoffs (queue-mediated, tool-invocation, or LLM-planned delegation). Does not apply to single-process synchronous call chains, nor to systems using only log-based correlation IDs without span parent linkage.

## Core Distinction

A trace *correlates* when its spans share a `trace_id`. A trace is *causally valid* only when every span's `parent_span_id` resolves to a span that (a) exists in the same trace, (b) has an end timestamp not later than the child's start timestamp minus clock-skew tolerance, and (c) was emitted by the process that actually invoked the child. Correlation is a labeling property; causality is a partial-order property. Handoffs routinely preserve the former while silently destroying the latter, because the handoff boundary is exactly where a planner agent reconstructs context from a serialized message rather than from an in-process span object. The failure is invisible in trace UIs that render by `trace_id` grouping alone.

## What Was Observed

Across three agent handoff topologies (queue-mediated worker pools, tool-call fan-out, and LLM-planned sub-agent delegation), the following structural signatures recurred:

1. **Orphan-parent rate spikes at handoff edges.** A span arriving at the receiving agent carried a `parent_span_id` referencing the *sending agent's planner span*, but the planner span had already been closed and exported before the child was created. This is not an error by itself, but it means the child's causal parent is a span whose duration does not enclose the child — a temporal impossibility that the backend accepted because it never validates parent-child intervals.

2. **Sibling adoption.** When the sending agent's span was absent from the exported batch (sampled out, dropped, or exported after a flush deadline), the receiving agent's span was reparented to the nearest surviving ancestor — often the trace root. The trace then *looks* well-formed (single root, no orphans) while the delegation edge is silently erased. The structural tell is a root span whose direct children include spans whose `service.name` and `agent.role` attributes indicate a deep delegation depth.

3. **Duplicate span IDs across retries.** Handoff retries (at-least-once delivery) re-executed the receiving agent's work with a freshly generated `span_id` but the same `parent_span_id` and identical `agent.input_hash`. Two siblings with byte-identical inputs and non-overlapping intervals indicate a retry, not a genuine parallel fan-out. Without this check, retries inflate fan-out metrics and corrupt critical-path computation.

4. **Clock inversion at the boundary.** Receiving agents on different hosts emitted child spans with `start_time` earlier than the parent's `start_time` by 40–900 ms, exceeding typical NTP skew. The cause was the receiving agent stamping span start from message-enqueue time (sender clock) while the parent span start was stamped from dequeue time (receiver clock). The partial order was inverted across the boundary even though both clocks were individually "correct."

## Condition of Validity

The causal-link invariant holds only under the conjunction:

- The `parent_span_id` is exported in the same batch or is guaranteed retrievable by the backend at query time (no sampling asymmetry between parent and child).
- Parent `end_time` ≤ child `start_time` + `max_clock_skew`, where `max_clock_skew` is a declared per-deployment constant, not a default of zero.
- The child span's `agent.handoff.from` attribute equals the parent span's `agent.id`, and this is asserted at emission, not reconstructed at query.
- Retry identity is carried explicitly (a `handoff.attempt` or idempotency key), so duplicate-input siblings are distinguishable from intentional fan-out.

If any conjunct fails, the trace is correlated but not causally valid, and any downstream computation that assumes the partial order (critical path, latency attribution, deadlock detection) is unsound.

## Decision Boundary

Do not attempt to *repair* causality at query time by reparenting orphans to the root or by "fixing" inverted timestamps. Reparenting fabricates a delegation edge that did not occur; timestamp clamping destroys the very signal that flags a clock-domain bug. The correct boundary is:

- **At emission:** if the parent span is not available in-process, emit the child with an explicit `handoff.parent_unavailable=true` attribute and the last-known `parent_span_id` rather than silently dropping the link or reparenting.
- **At ingestion:** reject or quarantine spans whose parent-child interval is inverted beyond `max_clock_skew`; do not silently accept.
- **At query:** treat missing-parent and inverted-interval spans as *unresolved*, excluded from critical-path math but retained for forensic inspection.

The decision to escalate (fail the handoff) versus annotate (degrade the trace) depends on whether the trace is load-bearing for correctness (e.g., used for exactly-once bookkeeping) or only for observability. Only the former justifies failing the handoff.

## Operational Effect

Enforcing the invariant at emission converts a class of silent observability corruption into a bounded, countable signal. In practice this moved orphan-parent detection from "discovered during incident postmortem" to a per-handoff assertion, and it surfaced a retry-induced fan-out inflation that had been misread as healthy parallelism for weeks. The cost is one attribute write and one interval comparison per handoff — negligible relative to the LLM calls the handoff wraps. The residual risk is that `max_clock_skew` becomes a fudge factor tuned to suppress alerts; it must be derived from measured host skew, not from the observed span inversions it is meant to catch.

```python
import time
from dataclasses import dataclass

MAX_CLOCK_SKEW_MS = 250.0


class CausalLinkError(RuntimeError):
    """Raised when a handoff span cannot be causally linked to its parent."""


@dataclass
class Span:
    trace_id: str
    span_id: str
    parent_span_id: str | None
    agent_id: str
    start_ms: float
    end_ms: float | None = None
    handoff_from: str | None = None


def link_handoff(child: Span, parent: Span | None) -> Span:
    if child.parent_span_id is None:
        raise CausalLinkError(
            f"span {child.span_id} has no parent_span_id at handoff boundary"
        )
    if parent is None:
        # Parent not in-process: annotate, never reparent to root.
        child.handoff_from = child.handoff_from or "unknown"
        return child
    if parent.span_id != child.parent_span_id:
        raise CausalLinkError(
            f"parent mismatch: child expects {child.parent_span_id}, "
            f"got {parent.span_id}"
        )
    if parent.end_ms is None:
        raise CausalLinkError(
            f"parent {parent.span_id} still open at child {child.span_id} emission"
        )
    if child.start_ms < parent.start_ms - MAX_CLOCK_SKEW_MS:
        raise CausalLinkError(
            f"clock inversion: child start {child.start_ms} precedes "
            f"parent start {parent.start_ms} beyond skew tolerance"
        )
    if parent.end_ms > child.start_ms + MAX_CLOCK_SKEW_MS:
        raise CausalLinkError(
            f"temporal impossibility: parent {parent.span_id} ends "
            f"{parent.end_ms} after child {child.span_id} starts {child.start_ms}"
        )
    if child.handoff_from != parent.agent_id:
        raise CausalLinkError(
            f"handoff provenance mismatch: child claims from "
            f"{child.handoff_from}, parent is {parent.agent_id}"
        )
    return child
```