# When must an instance stop trusting its own span correlation after memory mutation

**Date of Record:** 2026 November 14

**Scope:** Applies to agent observability pipelines that join GenAI inference spans to memory/identity events (vector-store writes, profile updates, credential rotation, session re-keying) within a single trace. Does not apply to stateless RAG lookups that emit no mutation event, nor to pipelines where memory and identity events are ingested by a separate trace store with no cross-join requirement.

## Core Distinction

The distinction that matters is between **span correlation** (the trace ID and parent/child linkage that says "this inference call happened inside this agent turn") and **entity correlation** (the assertion that the memory or identity state observed at span-emit time is the same state that existed when the span was *executed*). Observability pipelines routinely conflate these because both are carried in the same span attributes and both are joined on the same trace ID. Span correlation is immutable once emitted. Entity correlation is a claim about the world that can be invalidated retroactively by a mutation that lands between execution and export.

The non-obvious failure is this: a memory write or identity rotation that occurs *after* a GenAI span executes but *before* that span is exported will cause the exporter to attach the post-mutation entity version to a span that was produced under the pre-mutation version. The trace looks internally consistent — correct trace ID, correct parent, correct timestamps — but the entity correlation is silently wrong. There is no error, no dropped span, and no schema violation. The join succeeds and produces a false causal story.

## What Was Observed

In a pipeline where the agent process buffers spans in an in-memory batch exporter (flush interval ~5s) and memory mutations are emitted synchronously to the same collector, we observed the following sequence repeatedly under load:

1. Turn N executes a retrieval against memory namespace `user:4417`, version pointer `v12`.
2. The GenAI span is created with attributes `memory.namespace=user:4417` and `memory.version=v12` resolved lazily at export time from a live pointer object, not captured at span creation.
3. Before the batch flush, a consolidation job writes `v13` and updates the live pointer.
4. At flush, the exporter reads `memory.version` from the live pointer and stamps `v13` onto the span.
5. The collector joins the span to the mutation event for `v13` and concludes the inference read `v13`. It did not.

The downstream effect was a systematic bias in attribution: post-mutation versions accumulated more "reads" than they ever served, and pre-mutation versions appeared to have been read by spans that never touched them. An eval harness comparing "answers grounded in version X" against "version X was live" reported a 3.1% false-grounding rate that took two days to trace to lazy attribute resolution rather than to model behavior. The spans were not corrupt. They were *mis-attributed*, which is worse, because mis-attribution survives schema validation and passes every consistency check the pipeline had.

A second, sharper instance: identity rotation (API key and tenant re-key) between execution and export caused spans to be stamped with the *new* tenant identity while the request had actually been authorized under the old one. The audit join then attributed cross-tenant inference to the new tenant. This is the boundary where the failure stops being an observability bug and becomes a compliance defect.

## Condition of Validity

The entity correlation carried by a span is valid **if and only if** every entity-valued attribute on that span was captured by value at span execution time (or at the moment the guarded operation committed), and the span carries an explicit causal token — a version pointer, an etag, a monotonic revision — that the collector can use to detect retroactive invalidation.

It is invalid whenever any of the following holds:

- An entity attribute is resolved lazily from a mutable live object at export/flush time.
- The mutation event and the span are emitted through different paths with no shared ordering guarantee (e.g., synchronous mutation vs. batched span export).
- The collector joins on `(trace_id, entity_id)` without also joining on the captured revision token.
- Identity attributes (tenant, credential generation) are read from process-global state rather than from the request-scoped authorization context.

The condition is *not* satisfied merely because timestamps are monotonic. Wall-clock ordering between a batched flush and a synchronous mutation tells you nothing about which state the span observed; the span's `end_time` can precede the mutation's `timestamp` while the span still carries the post-mutation version, because the version was read at flush, not at end.

## Decision Boundary

An instance must **stop trusting its own span correlation** — i.e., must refuse to emit or accept an entity-correlated join — at the following boundary:

> If a span's entity attributes cannot be proven to have been captured before the first mutation to that entity that occurred after the span's execution began, the span must be emitted with entity correlation *suppressed* (attributes dropped or marked `entity.correlation=unverified`), not with best-effort values.

Concretely: capture entity attributes eagerly at span creation into immutable primitives; on export, compare the captured revision token against the collector's latest known revision for that entity. If the captured token is older than a mutation whose commit time falls within `[span.start, export_time]`, the join is ambiguous and must be downgraded. If the captured token is *newer* than the span's start (impossible under eager capture, diagnostic of lazy resolution), the span is corrupt and must be quarantined.

The boundary is asymmetric on purpose: an unverified span is a known unknown and can be sampled or re-derived; a confidently mis-attributed span is a silent lie that poisons every downstream aggregate it touches.

## Operational Effect

Enforcing this boundary converts a silent correctness failure into an explicit, countable one. In our pipeline, after switching to eager capture plus revision-token joins, roughly 0.4% of spans in mutation-heavy turns were downgraded to `entity.correlation=unverified` — these were turns where a consolidation landed mid-turn and the read genuinely could not be pinned to a single version. That 0.4% is the honest cost of the boundary. The 3.1% false-grounding rate collapsed to under 0.2%, and the residual was attributable to genuine mid-turn mutations rather than to exporter behavior. Cross-tenant mis-attribution went to zero because identity was captured from the request-scoped auth context and any span whose captured credential generation predated a rotation within its lifetime was quarantined rather than joined.

The operational lesson: entity correlation is a *claim*, and claims require evidence captured at the moment of observation. A trace ID proves the span happened; it proves nothing about what the span saw. Pipelines that treat the two as the same invariant will produce traces that are structurally perfect and semantically false, and the falseness will be invisible until an aggregate disagrees with reality.

```python
from dataclasses import dataclass
from typing import Optional


class EntityCorrelationError(RuntimeError):
    """Raised when a span's entity correlation cannot be trusted."""
    pass


@dataclass(frozen=True)
class CapturedEntity:
    entity_id: str
    revision: int
    captured_at: float  # monotonic seconds, at span creation


@dataclass(frozen=True)
class SpanRecord:
    trace_id: str
    span_id: str
    start_time: float
    end_time: float
    entity: Optional[CapturedEntity]


def verify_entity_correlation(
    span: SpanRecord,
    latest_revision: int,
    last_mutation_commit_time: float,
    export_time: float,
) -> str:
    """
    Decide whether the span's entity correlation is trustworthy.

    Returns 'verified' or 'unverified'. Raises EntityCorrelationError when the
    span is structurally corrupt (captured revision postdates execution, which
    can only happen under lazy resolution).
    """
    if span.entity is None:
        return "unverified"

    # Diagnostic: a captured revision must be observable no later than the
    # span's own start. A newer capture means the attribute was resolved late.
    if span.entity.captured_at > span.start_time + 1e-6:
        raise EntityCorrelationError(
            f"span {span.span_id} captured entity {span.entity.entity_id} "
            f"at {span.entity.captured_at} after start {span.start_time}: "
            f"lazy resolution detected, span is corrupt"
        )

    # A mutation committed during the span's lifetime, or between execution
    # and export, makes the join ambiguous if the captured revision is stale.
    mutation_in_window = last_mutation_commit_time >= span.start_time
    if mutation_in_window and span.entity.revision < latest_revision:
        return "unverified"

    # Captured revision newer than collector's latest is impossible under
    # eager capture and indicates a split-brain revision pointer.
    if span.entity.revision > latest_revision:
        raise EntityCorrelationError(
            f"span {span.span_id} claims revision {span.entity.revision} "
            f"but collector latest is {latest_revision}: revision pointer "
            f"divergence"
        )

    return "verified"


def emit_span(
    span: SpanRecord,
    latest_revision: int,
    last_mutation_commit_time: float,
    export_time: float,
) -> dict:
    status = verify_entity_correlation(
        span, latest_revision, last_mutation_commit_time, export_time
    )
    payload = {
        "trace_id": span.trace_id,
        "span_id": span.span_id,
        "entity.correlation": status,
    }
    if status == "verified":
        payload["entity.id"] = span.entity.entity_id
        payload["entity.revision"] = span.entity.revision
    # Unverified spans deliberately omit entity identity so no downstream
    # join can attach them to a specific version.
    return payload
```