# What happens when an instance mutates memory during a span it does not own: correlation loss across agent spans, memory writes, and compaction

**Date of Record:** 2026 November 13

**Scope:** Applies to GenAI observability pipelines that correlate agent execution spans with long-term memory mutation and compaction events using a shared trace/span identifier or causal parent link. Does not apply to systems where memory writes are strictly synchronous children of the span that triggered them and compaction is never concurrent with active spans.

## Core Distinction

There are two distinct notions of "ownership" in an agent observability pipeline, and they are routinely conflated:

1. **Span ownership** — the span ID stamped on an outbound memory mutation because the mutation was emitted from within that span's execution context (typically via context propagation of a trace context, e.g., an OpenTelemetry `ContextVar`).
2. **Causal ownership** — the span that is *actually responsible* for the mutation, which may be a different span than the one whose context happened to be active at the moment the write was flushed.

The failure mode arises when an asynchronous memory subsystem (a background consolidator, a write-behind queue, a post-turn reflection job) performs a mutation while the observability context of an *unrelated* span is active on the same worker. The mutation is then attributed to the wrong span — or worse, to a span belonging to a *different agent invocation entirely*. Compaction events compound this because they mutate the memory store in bulk, invalidating the very span-to-memory links that were recorded earlier.

The non-obvious invariant: **a memory mutation's recorded parent span must be the span that enqueued it, not the span that flushed it.** Pipelines that capture trace context at flush time violate this invariant silently and only under concurrency.

## What Was Observed

In a pipeline correlating agent turns (spans) with a vector-plus-graph memory store, the following sequence produced misattributed memory writes that survived downstream reconciliation:

- Agent invocation A begins span `S_A`. It enqueues a memory write `W_A` (a fact extracted from the turn) onto a write-behind queue. The queue item carries only the payload, not the trace context.
- Span `S_A` ends. The worker's context is cleared.
- Agent invocation B begins on the same worker, span `S_B`.
- The write-behind flusher drains `W_A` while `S_B` is the active span. The observability hook reads the *current* context and stamps `W_A` with `S_B`'s span ID.
- A compaction event `C` fires (triggered by store size threshold) and rewrites/relocates the memory records touched by `W_A`. The compaction span `S_C` is emitted as a sibling, but the pipeline's join key is `span_id`, so `W_A` now appears as a child of `S_B` and the compaction appears to have mutated memory "owned" by `S_B`.

Observed symptoms:

- Memory-write counts attributed per span were inflated for short spans that happened to overlap flush windows, and deflated for long spans that enqueued writes but ended before flush.
- A single compaction event appeared to "affect" dozens of unrelated spans because the join was on the wrong key.
- Replay-based evaluation (reconstructing an agent's memory state at span end) diverged from the live store, because the reconstruction used misattributed writes.
- The distortion was **load-dependent**: at low concurrency, flush almost always happened within the enqueuing span, so the bug was invisible in staging and in single-threaded tests.

## Condition of Validity

The misattribution occurs if and only if all of the following hold:

1. Memory mutations are **decoupled** from the span that caused them (write-behind queue, async consolidator, batched flusher, or post-turn job).
2. Trace context is captured at **flush/emit time** rather than at **enqueue time**.
3. More than one agent invocation can share a worker or event loop such that a different span's context is active during flush.
4. The correlation join downstream keys on `span_id` or `trace_id` alone, without an enqueue-time causal token.

If any condition fails, the pipeline is safe: synchronous writes (¬1) capture correct context; enqueue-time capture (¬2) is correct regardless of flush timing; single-invocation workers (¬3) never interleave; and a causal-token join (¬4) can detect the mismatch even when attribution is wrong.

Note the interaction with compaction: compaction does not *cause* misattribution, but it **amplifies and persists** it. Because compaction rewrites memory records and emits its own span, any downstream join that links spans to memory versions will attach the misattributed write to the compaction's output, making the error durable in derived datasets rather than transient in a log.

## Decision Boundary

Choose the remediation based on which of the four conditions you can actually change:

- **If you control the memory subsystem (¬1 or ¬2):** capture the trace context at enqueue time and carry it as an immutable field on the queue item. This is the correct fix; it makes the invariant hold structurally. Do not attempt to "fix" attribution by re-reading context at flush.
- **If you cannot change enqueue behavior (e.g., third-party memory store):** introduce a causal token — a monotonic enqueue sequence number plus the enqueuing span ID, written into the mutation payload — and join on that token downstream. Reject or quarantine mutations whose flush-time span differs from the enqueue-time span.
- **If compaction is concurrent with active spans:** treat compaction spans as *store-scoped*, not *span-scoped*. They must not be joined to individual agent spans by `span_id`; join them by store partition and time range, and mark affected memory versions as "compacted" rather than re-parenting them.
- **If neither fix is possible:** do not trust per-span memory attribution under concurrency; aggregate to invocation level and explicitly label attribution as approximate.

The boundary is: **enqueue-time context capture is mandatory whenever flush is asynchronous.** Any pipeline that cannot guarantee it must not present span-level memory attribution as authoritative.

## Operational Effect

After moving to enqueue-time capture and a causal-token join, the following changed measurably:

- Per-span memory-write counts became stable and load-invariant; the discrepancy between low- and high-concurrency runs dropped to zero.
- Compaction events stopped appearing as children of unrelated spans; they were re-scoped to store partitions, and the "affected spans" fan-out collapsed to the actual set of memory versions rewritten.
- Replay reconstruction matched the live store at span boundaries, restoring trust in offline evaluation.
- A new class of detectable error appeared: mutations whose causal token referenced a span that had already ended were surfaced as `OrphanedMutationError` rather than silently misattributed. This is the desired outcome — the failure became loud instead of corrupting the correlation graph.

The residual cost is a small amount of context plumbing on the enqueue path and a schema change to the queue item. That cost is paid once; the misattribution, left unfixed, is paid on every concurrent turn and is laundered into every derived dataset through compaction.

```python
from __future__ import annotations

from dataclasses import dataclass


class OrphanedMutationError(RuntimeError):
    """Raised when a memory mutation's flush-time span does not match its enqueue-time span."""


@dataclass(frozen=True)
class MutationEnvelope:
    mutation_id: str
    enqueue_span_id: str
    enqueue_trace_id: str
    payload: bytes


class MemoryWriteQueue:
    def __init__(self) -> None:
        self._items: list[MutationEnvelope] = []

    def enqueue(self, mutation_id: str, span_id: str, trace_id: str, payload: bytes) -> None:
        # Context is captured HERE, at enqueue time, not at flush time.
        self._items.append(
            MutationEnvelope(
                mutation_id=mutation_id,
                enqueue_span_id=span_id,
                enqueue_trace_id=trace_id,
                payload=payload,
            )
        )

    def flush(self, active_span_id: str) -> list[MutationEnvelope]:
        drained, self._items = self._items, []
        for item in drained:
            if item.enqueue_span_id != active_span_id:
                # The write is being flushed under a span that did not enqueue it.
                # Do NOT silently stamp it with the active span; surface the mismatch.
                raise OrphanedMutationError(
                    f"mutation {item.mutation_id!r} enqueued under span "
                    f"{item.enqueue_span_id!r} but flushed under span {active_span_id!r}"
                )
        return drained
```