# Why does an instance of an agent observability pipeline lose causal ordering between GenAI spans and memory-mutation events when context edits are recorded asynchronously

**Date of Record:** 2026 November 10

**Scope:** Applies to observability pipelines that ingest OpenTelemetry-style GenAI spans (LLM invocation, tool call, retrieval) alongside a separate memory-mutation/context-edit event stream (vector store writes, scratchpad rewrites, KV-cache eviction, prompt-context truncation). Does not apply to pipelines where memory mutations are emitted as child spans of the same trace synchronously within the instrumented process.

## Core Distinction

A GenAI span records *that a model was invoked and what it saw*. A memory-mutation event records *that the durable state the model depends on changed*. These are two different causal chains that frequently share a trace ID but not a clock. The pipeline is correct when it preserves the happens-before relation between a mutation and the span that observed it; it is broken when it preserves only wall-clock adjacency. The non-obvious failure is that a mutation emitted *after* a span's start but *before* its end can be attributed to the wrong span, and a mutation emitted *before* a span's start can be attributed to it at all only if the mutation's own causal parent is carried in the event, not inferred from timestamps.

## What Was Observed

In a deployment instrumenting an agent loop with a memory subsystem that flushed context edits on a background thread, the following sequence recurred:

1. Agent begins GenAI span `llm.invoke` (span_id `S1`) at `t=0`, reads context `C0`.
2. Tool call mutates memory (`memory.write` event `M1`) at `t=12ms`, emitted on the flush thread with its own timestamp but parented to the *root* span, not to `S1`.
3. `S1` ends at `t=40ms`.
4. Next `llm.invoke` (`S2`) starts at `t=41ms`, reads context `C1` (post-`M1`).

The pipeline's correlator, which joined mutations to spans by "mutation timestamp falls within span interval," attributed `M1` to `S1`. Downstream, the evaluator concluded that `S1`'s output was produced *after* the memory it depended on had changed, flagging a spurious "stale context" regression on 100% of turns. In reality `S1` read `C0` and `M1` landed between turns. The inverse also occurred: a synchronous mutation emitted at `t=-3ms` (just before `S1` started, same trace) was dropped by the windowing join because it fell outside `[start, end]`, producing silent gaps in the memory lineage.

The tell was that span durations and mutation counts were both correct; only the *edges* between them were wrong. No error was raised anywhere — the pipeline is lossy by construction under timestamp-only joins.

## Condition of Validity

Timestamp-interval correlation between GenAI spans and memory mutations is valid **only if** all of the following hold:

- Every mutation event carries an explicit causal parent (`trace_id` + `span_id` of the span that triggered it), not merely a timestamp.
- The clock used for mutation emission is the same monotonic source as span start/end, or clock skew is bounded below the shortest span interval.
- Mutations that occur between spans (turn boundaries) are represented as their own causal events, not orphaned by windowing.
- The memory subsystem's flush is either synchronous with the instrumented call or emits a completion marker that the span can await.

If any of these fail, the pipeline must treat mutation→span edges as *unverified* and refuse to compute derived metrics (staleness, context-dependency, cache-hit attribution) from them.

## Decision Boundary

The operator must choose between two postures at ingestion time:

- **Causal-join posture:** require `parent_span_id` on every mutation. If absent, emit the mutation to a `quarantine` stream and mark the trace as `lineage_incomplete`. Derived metrics are suppressed for that trace.
- **Interval-join posture:** accept timestamp-only correlation, but *only* when the pipeline can certify clock unity and span intervals are wider than the flush jitter. This posture must be explicitly opted into per-instrumentation, because it is the posture that silently misattributes.

Do not blend the two. A pipeline that uses causal joins when `parent_span_id` is present and falls back to interval joins when it is absent produces a lineage graph whose edge semantics change mid-trace — the worst possible outcome, because it looks complete.

## Operational Effect

Adopting the causal-join posture with quarantine surfaced that roughly 8% of mutation events in the deployment carried no parent — all from the background flush thread — which had previously been silently attributed. Suppressing derived staleness metrics for those traces removed the false-positive regression signal and replaced it with an honest `lineage_incomplete` rate. The cost is that some traces yield no staleness metric; the benefit is that the ones that do are trustworthy. The invariant to enforce: *a mutation is attributable to a span if and only if the mutation names that span as its causal parent; timestamps are for ordering within a causal chain, never for establishing the chain.*

```python
from __future__ import annotations

from dataclasses import dataclass


class LineageAttributionError(RuntimeError):
    """Raised when a memory-mutation event cannot be causally attributed to a span."""


@dataclass(frozen=True)
class GenAISpan:
    trace_id: str
    span_id: str
    start_ms: int
    end_ms: int


@dataclass(frozen=True)
class MemoryMutation:
    trace_id: str
    event_id: str
    timestamp_ms: int
    parent_span_id: str | None


def attribute_mutation(
    mutation: MemoryMutation,
    spans: dict[str, GenAISpan],
    *,
    allow_interval_fallback: bool = False,
    max_clock_jitter_ms: int = 0,
) -> GenAISpan:
    # Causal join: trust only an explicit parent.
    if mutation.parent_span_id is not None:
        parent = spans.get(mutation.parent_span_id)
        if parent is None:
            raise LineageAttributionError(
                f"mutation {mutation.event_id} names parent {mutation.parent_span_id} "
                f"not present in trace {mutation.trace_id}"
            )
        if parent.trace_id != mutation.trace_id:
            raise LineageAttributionError(
                f"mutation {mutation.event_id} parent {parent.span_id} belongs to "
                f"trace {parent.trace_id}, not {mutation.trace_id}"
            )
        return parent

    # No causal parent. Interval fallback is opt-in and must be certified.
    if not allow_interval_fallback:
        raise LineageAttributionError(
            f"mutation {mutation.event_id} has no parent_span_id; "
            f"refusing interval fallback (lineage_incomplete)"
        )

    candidates = [
        s
        for s in spans.values()
        if s.trace_id == mutation.trace_id
        and s.start_ms - max_clock_jitter_ms
        <= mutation.timestamp_ms
        <= s.end_ms + max_clock_jitter_ms
    ]
    if len(candidates) != 1:
        raise LineageAttributionError(
            f"mutation {mutation.event_id} matched {len(candidates)} spans by interval; "
            f"attribution is ambiguous"
        )
    return candidates[0]
```