# Which invariants prevent silent divergence between the compaction frontier and the retrieval index in long-horizon agent trajectories

**Date of Record:** 2026 November 12

**Scope:** Applies to agents that auto-compact trajectories by replacing spans of raw turns with lossy summaries while maintaining a retrieval index over the evicted raw spans for reconstitution-on-demand. Does not apply to lossless log compaction, to pure summarization without an eviction index, or to stateless context-window truncation.

## Core Distinction

Compaction and retrieval are two *independent* writers to the agent's effective state, and the invariant that binds them is not "summary is faithful to the span" but **"every fact the agent may later need is either present in the live summary chain or reachable via the retrieval index."** These are different properties. A summary can be locally faithful to the span it replaced and still drop a referent (an entity, an ID, a negation, a numeric bound) that the retrieval index cannot recover because the eviction boundary was drawn *after* the referent was already compressed into an intermediate summary rather than retained as raw text. The failure is a double-eviction: the fact is lossy in the summary layer and absent from the raw layer. The frontier of "what is summarized" and the frontier of "what is indexed" must not be allowed to cross.

## What Was Observed

On a 40-hour trajectory task (incident triage agent, ~14k turns, rolling compaction at a 70% context watermark, retrieval index over evicted raw spans), the agent correctly recalled a service name at hour 6, then at hour 31 asserted a *different* service name for the same incident. Investigation: the original span (hour 5–6) had been summarized at hour 8 into a summary that paraphrased the service name as "the upstream service." At hour 20 that summary was itself summarized (second-generation compaction) and the referent disappeared entirely. The raw span had been evicted from the retrieval index at hour 12 under a retention TTL that was keyed to *wall-clock age of eviction*, not to *whether the evicted span was still the sole carrier of a live fact*. Retrieval at hour 31 returned the hour-20 summary (referent-free), the agent pattern-matched to a plausible sibling service, and no error surfaced: the reconstitution path returned *something*, so the caller treated it as success. The divergence was silent because reconstitution had no notion of "this fact is no longer anchored anywhere."

The secondary observation: the agent's own confidence in the recalled name *increased* after compaction, because the lossy summary read as more declarative than the raw deliberation that preceded it. Summarization strips hedging. This is the mechanism by which lossy compaction converts uncertainty into false certainty without any single step being wrong.

## Condition of Validity

The failure requires all of: (1) multi-generation compaction, where a summary of a summary is possible; (2) a retrieval index whose retention is governed by a policy independent of fact-liveness (TTL, size cap, LRU); (3) reconstitution that returns a result without asserting that the requested fact is *anchored* — i.e., present in raw form or in a summary explicitly tagged as carrying it. Absent any one, the failure does not occur: single-generation compaction keeps the raw span reachable; a liveness-coupled index never evicts the sole carrier; anchored reconstitution raises instead of guessing.

## Decision Boundary

The boundary is: **a span may be evicted from the retrieval index only if every fact it uniquely carries is either (a) re-asserted verbatim in a live summary, or (b) marked as intentionally dropped with a tombstone that reconstitution treats as "unknown," not "absent."** In practice this means the compactor must emit, per summary, a *carried-facts manifest* (entities, IDs, negations, numeric bounds) and the index must refuse to drop the last raw or manifest-bearing carrier of any manifest entry still referenced by the live chain. When the manifest cannot be computed (unstructured span), the default must be *retain raw* — lossy compaction of an unmanifested span is the unsafe direction. The decision to compact is therefore not a context-budget decision; it is a *carrier-preservation* decision gated on the manifest.

## Operational Effect

Enforcing the manifest-and-anchor invariant raised steady-state context usage by ~9% (raw retention of unmanifested spans) and added one manifest pass per compaction. In exchange, cross-generation referent drift went to zero over a 200-hour replay, and the previously silent failure became a loud, catchable condition at reconstitution time. The operational lesson: the dangerous compaction bug is not a bad summary — it is a *successful* reconstitution of an unanchored fact. Make reconstitution fail closed.

```python
from dataclasses import dataclass, field
from typing import Optional


class UnanchoredReconstitutionError(RuntimeError):
    """Raised when a requested fact is neither in the live summary chain
    nor reachable as raw text / manifest-bearing carrier in the index."""
    pass


@dataclass
class FactManifest:
    # facts carried verbatim by a summary or raw span
    carried: set[str] = field(default_factory=set)


@dataclass
class Carrier:
    span_id: str
    is_raw: bool
    manifest: FactManifest
    generation: int  # 0 = raw, >0 = summary-of-summary


class CompactionIndex:
    def __init__(self) -> None:
        self.carriers: dict[str, Carrier] = {}
        self.live_chain: list[str] = []  # span_ids of active summaries
        self.tombstones: set[str] = set()

    def register(self, c: Carrier) -> None:
        self.carriers[c.span_id] = c

    def evict(self, span_id: str) -> None:
        # Refuse to evict the last carrier of any fact still referenced
        # by the live chain, unless a tombstone is explicitly recorded.
        target = self.carriers[span_id]
        for fact in target.manifest.carried:
            other_carriers = [
                c for sid, c in self.carriers.items()
                if sid != span_id and fact in c.manifest.carried
            ]
            if not other_carriers and fact not in self.tombstones:
                raise UnanchoredReconstitutionError(
                    f"refusing eviction of {span_id}: fact {fact!r} "
                    f"would become unanchored (no tombstone recorded)"
                )
        del self.carriers[span_id]

    def reconstitute(self, fact: str) -> Carrier:
        # Fail closed: return only if a carrier actually holds the fact.
        for sid in self.live_chain:
            c = self.carriers.get(sid)
            if c and fact in c.manifest.carried:
                return c
        for c in self.carriers.values():
            if fact in c.manifest.carried:
                return c
        if fact in self.tombstones:
            raise UnanchoredReconstitutionError(
                f"fact {fact!r} is tombstoned: intent was to drop it, "
                f"not to guess"
            )
        raise UnanchoredReconstitutionError(
            f"fact {fact!r} is unanchored: no raw span and no manifest "
            f"carrier holds it; reconstitution must not fabricate"
        )


def compact(index: CompactionIndex, span_ids: list[str], summary_id: str,
            summary_facts: set[str]) -> None:
    # Build the summary carrier first, then verify every fact in the
    # evicted spans is re-asserted or tombstoned before eviction.
    index.register(Carrier(summary_id, False, FactManifest(summary_facts), 1))
    index.live_chain.append(summary_id)
    dropped = set()
    for sid in span_ids:
        c = index.carriers[sid]
        dropped |= c.manifest.carried - summary_facts
    for fact in dropped:
        # Any fact the summary did not carry must be explicitly tombstoned
        # or the eviction below will raise.
        if fact not in index.tombstones:
            raise UnanchoredReconstitutionError(
                f"compaction dropped fact {fact!r} without a tombstone; "
                f"either carry it in the summary or mark it intentionally lost"
            )
    for sid in span_ids:
        index.evict(sid)
```