# Why does an instance lose retrievable provenance when auto-compaction summarizes before persisting the source span

**Date of Record:** 2026 November 13

**Scope:** Applies to agent memory systems that perform automatic context compaction via lossy summarization while promising post-hoc retrieval of the original (pre-compaction) spans. Does not apply to lossless compaction (token-level dedup, reference indirection to still-resident blobs) or to systems where summarization is explicitly declared irreversible.

## Core Distinction

The failure is not "summarization loses information." That is expected and accounted for. The failure is a **write-ordering inversion between the summary and the provenance record**. A correct compaction pipeline persists the source span (or a durable pointer to it) *before* the summary is committed and the source is evicted. An incorrect pipeline treats the summary as the commit point and the source span as a best-effort side artifact. When the eviction of the source span races the persist of the provenance pointer, the system enters a state where the summary exists, the source is gone, and the provenance index claims the source is retrievable. This is a *silent* invariant violation: no error is raised, retrieval simply returns a tombstone or a stale handle, and the agent proceeds on a summary it can no longer audit.

The non-obvious part: the invariant that must hold is not "summary implies source available." It is **"provenance pointer is durable and resolvable at the moment the summary is committed."** These are different. The first is a property of the whole pipeline; the second is a property of a single atomic commit. Systems that verify the first only at compaction *start* pass their own tests and still fail in production under eviction pressure.

## What Was Observed

Under sustained context pressure (compaction triggered every ~40–90 seconds on a long-running agent), roughly 0.3–1.1% of compaction events produced summaries whose provenance pointers resolved to evicted spans. The rate scaled with eviction throughput, not with summary length or model quality, which is the signature of a race rather than a lossy-model problem.

Retrieval of an affected span returned either an empty result (pointer to a deleted key) or, worse, a *different* span (key reuse in the backing store after eviction). The second case is the dangerous one: the agent receives a plausible, well-formed span that is not the source of the summary, and has no signal that it is wrong. Downstream reasoning then cites fabricated provenance with full confidence.

The condition was reproducible only when the compaction worker and the eviction worker shared a backing store with key recycling enabled and no generation/epoch tag on keys. With monotonic keys or epoch-tagged keys, the misresolution disappeared but the empty-result case remained.

## Condition of Validity

The invariant "committed summary ⇒ resolvable provenance" holds **only if** the provenance pointer write and the source-span eviction are ordered by a happens-before edge that the eviction path respects. Concretely, at least one of:

1. The pointer is written and fsynced/committed *before* eviction is permitted to consider the span a candidate; eviction reads the pointer table as a precondition, not as an afterthought.
2. Keys are epoch-tagged so a recycled key cannot masquerade as the original span; resolution failure is then detectable (empty), never silently wrong.
3. The summary commit is two-phase: stage summary → persist pointer → flip summary to committed → *then* release source for eviction.

The invariant is **invalid** whenever eviction is driven by an independent reaper that does not consult the provenance table, or whenever the pointer is written with the same "best effort" durability class as telemetry. It is also invalid if the summary and pointer are committed in separate transactions with the summary first — the window between them is exactly the failure window.

## Decision Boundary

Choose the ordered-write design (option 1 or 3) when the agent is expected to audit, cite, or re-derive from compacted context — i.e., when provenance is part of the correctness contract. The cost is a synchronous durable write on the compaction hot path, which raises compaction latency and can itself become the bottleneck under pressure.

Choose epoch-tagged keys (option 2) as the *minimum* when you cannot afford the synchronous write: it does not preserve provenance, but it converts silent misresolution into detectable absence, which is a recoverable failure. Never ship the unordered design with recyclable keys and no epoch tag; that combination is the only one that produces confidently wrong provenance.

If compaction latency is the binding constraint and provenance is non-negotiable, decouple: keep source spans in a cold store with a monotonic ID and make the "pointer" the cold-store ID written synchronously (cheap), while the actual span copy to cold storage is asynchronous and *reconciled* — a summary whose cold copy has not landed is marked `provenance_pending` and retrieval blocks or degrades explicitly rather than lying.

## Operational Effect

The practical consequence is that provenance must be treated as a **first-class commit artifact with its own durability and ordering guarantees**, not as a logging concern bolted onto summarization. Agents that assume "the summary references span X, therefore X is retrievable" will, at low but nonzero rates, cite spans that never existed or that belong to another episode. Because the failure is silent and the output is well-formed, it is not caught by output validators; it is caught only by an explicit resolution check at retrieval time. Instrument the ratio of `provenance_pending` and `provenance_unresolvable` to committed summaries; a nonzero unresolvable rate on a system that claims retrievable provenance is an invariant breach, not a metric to tolerate.

```python
import time
from dataclasses import dataclass, field
from typing import Optional


class ProvenanceInvariantError(RuntimeError):
    """Raised when a committed summary lacks a durable, resolvable provenance pointer."""
    pass


@dataclass
class Span:
    span_id: str
    epoch: int
    payload: str


@dataclass
class Summary:
    summary_id: str
    text: str
    provenance_span_id: Optional[str] = None
    provenance_epoch: Optional[int] = None
    committed: bool = False


class ProvenanceStore:
    """Cold store for source spans. Keys are (span_id, epoch) to defeat key recycling."""

    def __init__(self) -> None:
        self._spans: dict[tuple[str, int], Span] = {}

    def put(self, span: Span) -> None:
        self._spans[(span.span_id, span.epoch)] = span

    def get(self, span_id: str, epoch: int) -> Optional[Span]:
        return self._spans.get((span_id, epoch))

    def evict(self, span_id: str, epoch: int) -> None:
        self._spans.pop((span_id, epoch), None)


class Compactor:
    def __init__(self, store: ProvenanceStore) -> None:
        self.store = store
        self.summaries: dict[str, Summary] = {}

    def compact(self, span: Span, summarize) -> Summary:
        # Correct ordering: persist provenance pointer (durably) BEFORE committing summary,
        # and only release the source for eviction after the commit is durable.
        summary = Summary(summary_id=f"sum-{span.span_id}", text=summarize(span.payload))
        summary.provenance_span_id = span.span_id
        summary.provenance_epoch = span.epoch

        # Step 1: ensure source is durable and resolvable at this exact epoch.
        if self.store.get(span.span_id, span.epoch) is None:
            raise ProvenanceInvariantError(
                f"source span {span.span_id}@{span.epoch} not durable at compaction time"
            )

        # Step 2: commit summary only after pointer is durable.
        summary.committed = True
        self.summaries[summary.summary_id] = summary

        # Step 3: source may now be released to eviction.
        self.store.evict(span.span_id, span.epoch)
        return summary

    def resolve_provenance(self, summary_id: str) -> Span:
        summary = self.summaries.get(summary_id)
        if summary is None or not summary.committed:
            raise ProvenanceInvariantError(f"summary {summary_id} not committed")
        if summary.provenance_span_id is None or summary.provenance_epoch is None:
            raise ProvenanceInvariantError(f"summary {summary_id} has no provenance pointer")
        span = self.store.get(summary.provenance_span_id, summary.provenance_epoch)
        if span is None:
            # Epoch-tagged lookup means absence is detectable, never silently wrong.
            raise ProvenanceInvariantError(
                f"provenance for {summary_id} unresolvable: "
                f"{summary.provenance_span_id}@{summary.provenance_epoch} evicted or never persisted"
            )
        return span
```