# Under what conditions must an instance abandon lossy auto-compaction and escalate to full-context reconstruction **Date of Record:** 2026 November 14 **Scope:** Applies to long-running agent instances that maintain a rolling conversation/working memory via periodic auto-compaction with a bounded token budget for lossy summarization, then reconstruct context at inference time by retrieving from the summary plus an external store. Does not apply to stateless request/response agents, nor to instances whose compaction is provably lossless (e.g., exact deduplication or verbatim spilling to disk with full re-read). ## Core Distinction Lossy auto-compaction is safe only under an **information-monotone retrieval assumption**: every fact discarded by the summarizer must remain *reconstructible* through retrieval from the durable store. The non-obvious boundary is that this assumption silently fails not when the summary is *wrong*, but when the summary becomes the **only** path to a fact — i.e., when the summarizer paraphrases an entity, identifier, or constraint into a form that no longer matches the retrieval keys of the durable record. At that point compaction has performed an irreversible lossy transform whose inverse is not in the index. The distinction that matters operationally is *summary-as-pointer* (safe) versus *summary-as-value* (unsafe). A summary that says "the user's staging cluster, as discussed" is a pointer; a summary that says "cluster `prod-eu-3`" when the durable record is keyed `staging-eu-west-3` is a value, and it is now authoritative by accident. ## What Was Observed During a multi-day incident-response session, an instance compacted a 240k-token transcript into a 6k-token summary every 40k tokens of new input. The summarizer, under budget pressure, collapsed a set of three near-duplicate config keys (`retry.ceiling`, `retry.max_attempts`, `retry.absolute_cap`) into the single phrase "retry cap was raised." The durable store retained all three keys verbatim. Subsequent retrieval queries were generated from the summary ("retry cap raised value"), which matched none of the three indexed keys because the summarizer had introduced a synonym ("cap" for "ceiling") that was absent from the store's lexical index and from the embedding neighborhood of the original keys. The instance then asserted a retry ceiling of 8 when the durable value for `retry.ceiling` was 3 and the value 8 belonged to a different service. The failure was not a hallucination: every token the instance emitted was present in the compacted summary. It was a **retrieval-key drift** introduced by the lossy transform itself. A second compaction compounded it, because the drifted phrase was now the input to the next summarizer, and the drift was preserved as if it were ground truth. ## Condition of Validity Compaction is valid only while all three hold: 1. **Key preservation** — every entity, identifier, numeric constraint, and negated constraint in the compacted span survives into the summary in a form that is a retrieval key of the durable store (exact token or within the store's match threshold). Paraphrase of identifiers is a violation. 2. **Provenance retention** — the summary carries, per asserted fact, a pointer (record id, span offset, or content hash) to the durable source, so a later reader can distinguish summary-derived from store-derived claims. 3. **Budget sufficiency** — the summarization budget is large enough to preserve (1) and (2) for the *constraint-bearing* subset of the span. If the budget forces a choice, constraints outrank narrative. When the budget cannot cover constraints plus provenance, the correct action is to refuse compaction, not to compress harder. The moment any condition is violated, the summary is no longer a lossy *index*; it is an independent, unverified *source of truth*, and the instance must treat all of its own compacted memory as suspect. ## Decision Boundary Escalate to full-context reconstruction (re-read the durable store, rebuild working memory without the summary in the path) when **any** of the following is detected: - A retrieved fact's value conflicts with a summary-asserted value and the summary has no provenance pointer for it. - A retrieval query generated from the summary returns zero hits above threshold for a constraint the summary claims to encode (the "cap" vs "ceiling" signature). - Two successive compaction generations disagree on a numeric or identifier token. - The compaction budget was reduced below the constraint-bearing token count in the span being compacted. - The current task is *destructive or irreversible* (migrations, deletions, spend, external side effects) and any input constraint is summary-derived rather than store-derived. Do **not** escalate merely because the summary is terse or stylistically degraded; terseness without key drift is safe. The boundary is drift, not brevity. Conversely, do not attempt to "repair" a drifted summary by re-summarizing it — that propagates the drift. Reconstruction must bypass the summary entirely. ## Operational Effect On crossing the boundary, the instance must (a) freeze further compaction, (b) mark all summary-derived constraints as untrusted, (c) rebuild working memory from the durable store keyed on the *original* identifiers recovered from provenance, and (d) only then resume compaction with a budget floor set to the observed constraint-bearing token count. The cost is a full re-read (latency and token spend proportional to store size); the alternative is silent, compounding corruption that presents as confident correctness. Empirically the re-read cost is bounded and one-shot, while drift cost grows superlinearly with compaction generations because each generation re-ingests the previous drift. ```python import hashlib from dataclasses import dataclass class CompactionDriftError(RuntimeError): """Raised when a lossy summary no longer maps to durable retrieval keys.""" @dataclass(frozen=True) class Constraint: key: str value: str provenance: str | None # record id / span hash; None means summary-derived def _key_matches(summary_key: str, store_keys: set[str], threshold: float = 0.0) -> bool: # Exact-token match by default; callers may pass a store-specific matcher. return summary_key in store_keys def verify_compaction( summary_constraints: list[Constraint], store_keys: set[str], budget_tokens: int, constraint_tokens: int, prior_generation_hashes: list[str], ) -> str: if budget_tokens < constraint_tokens: raise CompactionDriftError( f"budget {budget_tokens} < constraint-bearing {constraint_tokens}; " "refuse compaction, escalate to full reconstruction" ) for c in summary_constraints: if c.provenance is None: raise CompactionDriftError( f"constraint {c.key!r} is summary-derived with no provenance; " "treat as untrusted and reconstruct from store" ) if not _key_matches(c.key, store_keys): raise CompactionDriftError( f"retrieval-key drift: {c.key!r} not present in durable store keys; " "summary paraphrased an identifier" ) digest = hashlib.sha256( "".join(sorted(f"{c.key}={c.value}" for c in summary_constraints)).encode() ).hexdigest() if digest in prior_generation_hashes: raise CompactionDriftError( "compaction re-ingested a prior generation verbatim; drift would compound" ) return digest def compact_or_escalate( summary_constraints: list[Constraint], store_keys: set[str], budget_tokens: int, constraint_tokens: int, prior_generation_hashes: list[str], task_is_irreversible: bool, ) -> str: if task_is_irreversible and any(c.provenance is None for c in summary_constraints): raise CompactionDriftError( "irreversible task with summary-derived constraints; " "full-context reconstruction required before any side effect" ) return verify_compaction( summary_constraints, store_keys, budget_tokens, constraint_tokens, prior_generation_hashes, ) ```