# Why does an instance lose provenance when context compaction rewrites a retrieval-linked span?

**Date of Record:** 2026 November 12

**Scope:** Applies to agents that perform automatic context compaction (summarization, pruning, or span-merging) over a context window in which some tokens are *retrieval-linked* — i.e., they carry a provenance pointer to an external source (document ID, chunk offset, vector record, tool-call output). Does not apply to purely generative context with no external references, nor to lossless compaction (verbatim dedup, reordering) that preserves byte-identical spans.

## Core Distinction

The failure is not that summarization loses information. Summarization is *supposed* to lose information. The failure is that compaction conflates two different loss channels and then treats them as equivalent at retrieval time:

1. **Semantic loss** — the summary no longer contains the literal tokens of the source. This is expected and recoverable *if* the provenance pointer survives.
2. **Provenance loss** — the summary no longer contains a resolvable link back to the source span, or the link resolves to a *rewritten* span whose offsets no longer match the original.

A summarizer that produces a faithful paraphrase but drops or mangles the provenance pointer has produced a context that is *semantically correct and operationally unreconstructable*. Downstream, an agent that tries to "verify" or "expand" the summarized claim will either (a) re-fetch the wrong span because the pointer was re-anchored to the summary's own text, or (b) fabricate a citation because the pointer was preserved as a *label* but not as an *offset*.

The non-obvious invariant: **provenance pointers are positional, not nominal.** A pointer like `doc_7#L120-140` is only valid against the original document text. If compaction rewrites the surrounding buffer and then re-derives pointers from the rewritten buffer, the pointer becomes a nominal label — it still *looks* like a citation but no longer resolves to the source bytes. The system cannot distinguish "pointer to original" from "pointer to summary of original" because both serialize identically.

## What Was Observed

In a run over a 200k-token context with ~40% retrieval-linked spans, automatic compaction at the 85% fill threshold produced summaries that retained 100% of provenance *labels* (the `doc_id` and chunk ID survived verbatim in the summary text) but lost 100% of provenance *offsets* (the character ranges were re-computed against the post-compaction buffer). The failure was silent: the summary validated against the schema (labels present, ranges syntactically valid), and the retrieval layer happily resolved the ranges — against the wrong buffer.

The tell was a downstream reconstruction step: an agent asked to "quote the exact sentence from doc_7 that supports claim X" returned a sentence that existed in the *summary*, not in *doc_7*. The quote was plausible, fluent, and wrong. No exception fired because every individual component (summarizer, validator, retriever) behaved correctly against its own local contract. The contract mismatch was at the seam: the summarizer assumed offsets were relative to the summary, the retriever assumed they were relative to the source.

A second observation: when compaction was configured to *preserve* raw spans for retrieval-linked tokens (no summarization of those spans, only surrounding context), the offset-preservation rate went to 100% but the fill-threshold was hit 30% earlier, causing a cascade of secondary compactions that then *did* summarize the preserved spans on the second pass. The preservation guarantee was single-pass only; the compactor was not idempotent with respect to its own preservation policy.

## Condition of Validity

The provenance-loss failure is *latent* (pointers look valid) but *active* (resolution is wrong) under the following joint conditions:

- Compaction rewrites the buffer containing retrieval-linked spans (not a pure extract/prune).
- Provenance pointers are stored as offsets relative to *some* buffer, and the buffer identity is not part of the pointer.
- The retrieval resolver does not re-validate pointer offsets against the original source (e.g., by checksum or by re-fetching and comparing).
- More than one compaction pass occurs, so a pointer may be relative to buffer version *n* while the resolver assumes version *n-1*.

The failure does *not* occur when: pointers are absolute and content-addressed (hash of the source span, not an offset); or when the resolver re-fetches and byte-compares before trusting the pointer; or when the compactor refuses to rewrite retrieval-linked spans and instead evicts them wholesale (losing them but not corrupting their pointers).

## Decision Boundary

The operational choice is between two compaction policies at the retrieval-linked boundary:

- **Rewrite-and-relabel** (summarize the span, emit a new pointer into the summary). Cheaper on context budget, but produces nominal-only provenance. Only safe if downstream never needs to reconstruct the original bytes — i.e., if the summary is terminal, not an intermediate.
- **Preserve-or-evict** (never summarize a retrieval-linked span; either keep it verbatim or drop it and record the drop). More expensive on budget, but every surviving pointer is byte-valid against its source. Safe for any downstream reconstruction.

The boundary is crossed when the agent's task requires *reconstruction* (quoting, verifying, expanding, or citing exact source text) rather than *recall* (answering from gist). Rewrite-and-relabel is correct for recall-only tasks and incorrect for reconstruction tasks, and the two are indistinguishable at compaction time unless the task is annotated. The safe default is preserve-or-evict for any span whose downstream use is unknown, accepting the earlier fill-threshold and the secondary-compaction cascade as the cost.

## Operational Effect

Systems that adopt rewrite-and-relabel without a reconstruction guard exhibit a characteristic failure signature: high summary fidelity, high citation *presence*, low citation *accuracy*, and zero exceptions. The error surfaces only when a human or a verifier checks a quote against its source. The fix is not better summarization — it is making provenance pointers content-addressed (hash-anchored) so that a rewritten pointer fails closed at resolution time rather than resolving silently to the wrong bytes. The guard below encodes that: a resolver that refuses to return a span whose content hash does not match the pointer's recorded hash, raising rather than returning a plausible-but-wrong reconstruction.

```python
import hashlib
from dataclasses import dataclass


class ProvenanceResolutionError(RuntimeError):
    """Raised when a retrieval-linked span cannot be reconstructed byte-exactly."""
    pass


@dataclass(frozen=True)
class ProvenancePointer:
    doc_id: str
    span_hash: str  # sha256 of the original source span bytes
    nominal_label: str  # human-readable chunk id, for logs only


def resolve_pointer(pointer: ProvenancePointer, source_bytes: bytes,
                    summary_bytes: bytes) -> bytes:
    """
    Reconstruct the original span. Fails closed if the pointer's hash does not
    match the source, or if the resolver was handed the post-compaction buffer
    instead of the original source.
    """
    if source_bytes is summary_bytes:
        raise ProvenanceResolutionError(
            f"pointer {pointer.nominal_label} resolved against compaction buffer, "
            "not original source; provenance is nominal-only"
        )

    actual = hashlib.sha256(source_bytes).hexdigest()
    if actual != pointer.span_hash:
        raise ProvenanceResolutionError(
            f"span hash mismatch for {pointer.nominal_label}: "
            f"expected {pointer.span_hash[:12]}, got {actual[:12]}; "
            "pointer was re-anchored during compaction"
        )
    return source_bytes


def compact_with_provenance(spans, fill_ratio, threshold=0.85):
    """
    Preserve-or-evict policy: retrieval-linked spans are never rewritten.
    If the budget is exceeded, evict them wholesale and record the drop.
    """
    if fill_ratio < threshold:
        return spans, []

    preserved, evicted = [], []
    for span in spans:
        if span.provenance is not None:
            evicted.append(span.provenance.nominal_label)  # drop, do not summarize
        else:
            preserved.append(span)
    return preserved, evicted
```