# What happens when an instance loses its trace-parent during a tool invocation: orphaned memory operations and the correlation collapse boundary

**Date of Record:** 2026 November 7

**Scope:** Applies to agent runtimes that correlate LLM reasoning steps, memory reads/writes, and external tool calls under a single distributed trace identifier propagated via context (W3C `traceparent`, OpenTelemetry baggage, or equivalent). Does not apply to single-process agents whose memory and tool calls execute synchronously within one span with no context hand-off.

## Core Distinction

There is a difference between *trace loss* and *correlation collapse*. Trace loss means a span was never emitted — the observability backend simply has no record. Correlation collapse is subtler and far more dangerous: every span *is* emitted, every memory operation and tool invocation carries a valid span ID and timestamp, but the **parent-child link** between the agent's reasoning span and the downstream memory/tool spans is severed. The spans exist as disconnected roots in the trace graph.

Operators routinely conflate these. A missing span is loud (dashboards show gaps). A collapsed correlation is silent: span counts look healthy, latency histograms are populated, and error rates are normal — but the causal chain that lets you answer "which memory write did this tool call depend on?" is gone. The data is present and the *relationship* is absent.

## What Was Observed

During a multi-hour agent session with a vector-memory backend and three tool integrations, an instance crossed a context-propagation boundary inside an async tool executor. The executor used a thread-pool offload (`run_in_executor`) to perform a blocking memory write. The runtime's context-variable carrier was not copied into the worker thread. The memory-write span was created correctly — it inherited the *ambient* trace ID from a process-global default rather than the request-scoped one — so its `trace_id` field matched the session, but its `parent_span_id` was null.

Observed consequences across the session:

1. Memory writes appeared as root spans. The trace graph rendered them as independent traces of depth 1, disconnected from the agent turn that produced them.
2. Tool invocations that *read* from memory were attributed to the agent's reasoning span correctly (they ran on the main event loop), creating an asymmetry: reads were children of the turn, writes were orphans. Any causal query like "show the memory write that preceded this tool read" returned nothing.
3. Because the orphaned write spans shared the session `trace_id` but not the parent, downstream sampling logic that sampled per-root treated each memory write as an independent sampling decision. Roughly 40% of memory-write spans were dropped by the tail sampler while the corresponding tool-read spans were retained, producing a trace that *looked* complete but was missing half its memory history.
4. Retry storms amplified it: the executor retried the blocking write three times on timeout, producing three orphaned root spans per logical write, all with identical attributes. Deduplication keyed on `(trace_id, span_id)` could not collapse them because each retry minted a fresh span ID.

The invariant that was violated: **every span emitted on behalf of an agent turn must share the turn's `trace_id` AND a non-null `parent_span_id` chain back to the turn root.** The system enforced the first half of the invariant (trace_id propagation via ambient default) and silently failed the second (parent linkage via context-variable copy). Partial invariant enforcement is worse than none, because it defeats the integrity checks that would otherwise catch the break.

## Condition of Validity

The correlation guarantee holds only when all of the following are true simultaneously:

- Context (trace ID *and* current span ID) is propagated by explicit copy across every concurrency boundary: thread-pool offloads, `asyncio.to_thread`, subprocess spawns, and message-queue hand-offs. Ambient/global defaults are **not** valid propagation and must be treated as absent context.
- The span is created *from* the propagated parent context, not from a process-global tracer default. A tracer configured with a fallback default context will synthesize plausible-looking but wrong parents.
- Memory operations and tool invocations are emitted within the lifetime of the turn span; a span that outlives its parent (fire-and-forget writes flushed after the turn closes) cannot be parented and must be explicitly re-attached or marked as a deferred operation with a recorded `turn_id` attribute.
- Sampling decisions are made per trace root, not per span, and the sampler must not sample orphaned roots independently.

When any of these fails, the trace graph is not merely incomplete — it is *misleading*, which is the failure mode this record addresses.

## Decision Boundary

Do not attempt to reconstruct correlation from timestamps and shared attributes after the fact. Timestamp-based join ("the memory write whose timestamp is closest before this tool read") is unreliable under concurrent turns and produces confidently wrong causal chains — worse than admitting the link is unknown. The decision boundary is:

- If `parent_span_id` is null on a span that carries a `turn_id` attribute, treat the correlation as **broken, not missing**. Emit a `correlation_collapse` counter and refuse to serve causal queries over that trace rather than guessing.
- If a tool invocation's span is a child of the turn but its preceding memory write is an orphaned root, the read is valid but its dependency is unverifiable. Downstream reasoning that consumes that read must be flagged as `provenance_unverified`.
- Only when the full parent chain from turn root to memory write to tool read is intact may the runtime assert causal ordering. Anything less is a degraded mode that must be surfaced, not smoothed over.

The boundary is: **correlation is either structurally guaranteed by propagation or explicitly declared absent. It is never inferred.**

## Operational Effect

After the incident, the runtime was changed to fail closed on context propagation. Any worker-thread span creation without an explicitly copied parent context now raises rather than silently parenting to the global default. The tail sampler was reconfigured to sample per trace root and to force-retain any span carrying a `turn_id`, so orphaned writes are visible as anomalies instead of being probabilistically dropped. Memory-write retries were made idempotent with a stable `logical_write_id` so duplicate orphaned roots collapse. The observable result: orphaned memory spans dropped to zero, and the `correlation_collapse` counter became a leading indicator that fires on the *first* broken propagation rather than after a session's worth of silent data loss. The key operational lesson is that partial propagation — correct trace ID, absent parent — is the worst configuration, because it passes every superficial health check while quietly destroying the causal graph the observability system exists to provide.

```python
from __future__ import annotations

from dataclasses import dataclass
from typing import Optional


class CorrelationCollapseError(RuntimeError):
    """Raised when a span cannot be structurally linked to its turn root."""


@dataclass(frozen=True)
class SpanContext:
    trace_id: str
    span_id: str
    parent_span_id: Optional[str]
    turn_id: Optional[str]


def emit_memory_write_span(
    propagated: Optional[SpanContext],
    logical_write_id: str,
) -> SpanContext:
    """Create a memory-write span that MUST be parented to the active turn span.

    Propagation across a concurrency boundary (thread pool, subprocess) is only
    valid via explicit copy. An absent context is NOT a license to fall back to
    a process-global default; that produces an orphaned root span which silently
    breaks causal correlation.
    """
    if propagated is None:
        raise CorrelationCollapseError(
            f"no propagated span context for memory write {logical_write_id!r}; "
            "refusing to parent to ambient default (would create orphaned root)"
        )
    if propagated.parent_span_id is None and propagated.turn_id is None:
        raise CorrelationCollapseError(
            f"propagated context for {logical_write_id!r} has neither parent nor "
            "turn_id; causal link to turn root is unverifiable"
        )

    # New span is a child of the propagated span, carrying the logical write id
    # so idempotent retries collapse on (trace_id, logical_write_id).
    return SpanContext(
        trace_id=propagated.trace_id,
        span_id=f"mem-{logical_write_id}",
        parent_span_id=propagated.span_id,
        turn_id=propagated.turn_id,
    )


def assert_causal_chain(
    turn_root: SpanContext,
    memory_write: SpanContext,
    tool_read: SpanContext,
) -> None:
    """Assert turn_root -> memory_write -> tool_read is structurally intact."""
    if memory_write.parent_span_id != turn_root.span_id:
        raise CorrelationCollapseError(
            f"memory write {memory_write.span_id!r} is not a child of turn root "
            f"{turn_root.span_id!r}; correlation is broken, not missing"
        )
    if tool_read.parent_span_id != memory_write.span_id:
        raise CorrelationCollapseError(
            f"tool read {tool_read.span_id!r} does not descend from memory write "
            f"{memory_write.span_id!r}; provenance is unverified"
        )
```