# How does an instance detect summarization-induced invariant collapse in auto-compaction of long-horizon agent transcripts

**Date of Record:** 2026 November 8

**Scope:** Applies to agents that persist transcripts across context boundaries and trigger auto-compaction (lossy summarization) when token budgets are exceeded. Does not apply to lossless compaction (deduplication, reordering, reference-by-id) or to single-turn sessions where no cross-boundary state is carried.

## Core Distinction

There is a difference between *compaction that preserves the working set* and *compaction that preserves the narrative*. A summarizer can produce fluent, plausible summaries that score well on ROUGE-like metrics while silently dropping the specific tokens that bind the agent to external commitments: an exact file path, a numeric threshold, a tool-call argument, a promise made to the user, or the identity of a pending side effect. The failure is not "the summary is wrong." The failure is that the summary is *locally coherent and globally falsified* — it retains the shape of the transcript but loses the invariants the transcript was carrying.

The non-obvious boundary: compaction is safe iff the set of **obligations** (tokens that a later turn will be evaluated against) is disjoint from the set of **discarded** spans. Most compaction policies enforce a token-budget constraint and a recency constraint, but no obligation-preservation constraint. The invariant that collapses is *referential integrity across the compaction boundary*: a later turn refers to `X`, but `X` was elided and the summary substituted a paraphrase `X'` where `X' ≠ X`.

## What Was Observed

In a long-horizon session (≈180 turns) with a 96k-token budget and a compaction trigger at 80%, the following sequence recurred:

1. Turn 41: user states "use the staging bucket `s3://acme-stage-07/`, never prod." This is an obligation with two parts: a positive literal and a negative literal.
2. Turn 63: compaction fires. Summary reads: "User prefers staging environment over production." The literal `acme-stage-07` is dropped; the negative constraint is softened to a preference.
3. Turn 97: agent constructs a deploy command. It has no access to `acme-stage-07`, so it synthesizes `acme-staging` from the summary's paraphrase.
4. Turn 98: tool call fails with `NoSuchBucket`. Agent retries with `acme-prod` inferred from the phrase "over production" as a fallback.
5. Turn 99: the *negative* invariant is violated. The summary said "prefers staging over production," which the agent read as a ranking, not a prohibition.

The diagnostic signature: the agent's later turns are *internally consistent with the summary* and *externally inconsistent with the transcript*. A naive monitor that checks "does turn N agree with turn N−1" passes. A monitor that checks "does turn N agree with the pre-compaction obligation ledger" fails.

A second observed mode: **obligation drift under repeated compaction**. After the third compaction, a numeric threshold (`retry ≤ 3`) had become "a few retries," then "retries should be limited." Each compaction was individually defensible; the composition was not. This is the lossy-summarization analog of generational loss in re-encoding — errors compound multiplicatively, not additively, because each summary is generated from the previous summary, not from the original.

## Condition of Validity

The invariant "compaction preserves obligations" holds only under all of:

- **Obligation extraction is total.** Every span that a future turn could be evaluated against is tagged at write time, not at compaction time. Compaction-time extraction is too late: the summarizer sees only the window it is compacting and cannot know which spans are load-bearing for turns it has not seen.
- **Obligations are carried by reference, not by paraphrase.** The summary may describe context, but obligation tokens (identifiers, literals, thresholds, negations) must be copied verbatim into a side-channel ledger that survives compaction unmodified.
- **Negations are first-class.** A summarizer that treats "never X" as "prefers not-X" has inverted a hard constraint into a soft preference. This is the single most common collapse.
- **Compaction is idempotent on the ledger.** Compacting twice must not degrade the ledger. If the ledger is itself summarized, the invariant is void.
- **The agent re-reads the ledger before any side-effecting tool call**, not before generation. Generation-time recall is unreliable; call-time gating is enforceable.

Outside these conditions — e.g., exploratory sessions with no external side effects, or sessions where the user re-states constraints each turn — the stricter policy is unnecessary overhead.

## Decision Boundary

Detect collapse by comparing the *obligation ledger* against *post-compaction references*, not by comparing summaries to transcripts. The decision rule:

- If a post-compaction turn references an identifier, literal, or threshold that is **not** in the ledger and **not** derivable from the ledger by exact match → **halt and rehydrate**. Do not let the model "reconstruct" it; rehydration means fetching the original span from cold storage.
- If a post-compaction turn takes a side-effecting action whose preconditions include a **negated** obligation → **block**, regardless of summary confidence.
- If the ledger's obligation count decreases across a compaction event without an explicit user retraction → **invariant violation**, treat as data loss.
- If the agent's plan cites a paraphrase (`acme-staging`) where the ledger holds a literal (`acme-stage-07`) → **soft failure**, log and require exact-literal substitution before execution.

The boundary is *not* "summary quality." A high-quality summary of a transcript with a dropped literal is still a violation. The boundary is set membership: `referenced_tokens ⊆ ledger_tokens ∪ derivable_tokens`.

## Operational Effect

When the boundary is enforced, compaction becomes a *lossy compression of narrative with lossless compression of obligations*. The practical effects:

- Context budget is spent on the ledger (small, bounded) plus the summary (large, unbounded), rather than on the summary alone. The ledger is the cheap insurance.
- Rehydration events become rare and explicit, rather than silent and diffuse. An agent that rehydrates on a ledger miss spends one tool call; an agent that hallucinates a bucket name spends a failed deploy.
- Multi-compaction sessions remain stable because the ledger is not summarized. Generational loss is confined to narrative, which is allowed to drift.
- The failure mode shifts from *silent invariant collapse* to *loud ledger miss*, which is the correct direction: a boundary failure you can observe is an operational event; one you cannot is an incident.

The residual risk is ledger scope creep — obligations accumulate and the ledger becomes a second transcript. Bound it: obligations expire when the user retracts them or when the referenced resource is destroyed, and the ledger is capped at a fixed fraction (observed working value: 8–12%) of the context budget.

```python
from dataclasses import dataclass, field
from typing import Iterable


class ObligationIntegrityError(RuntimeError):
    """Raised when a post-compaction reference cannot be bound to the obligation ledger."""


@dataclass(frozen=True)
class Obligation:
    literal: str
    polarity: str  # "require" | "forbid"
    origin_turn: int


@dataclass
class ObligationLedger:
    obligations: dict[str, Obligation] = field(default_factory=dict)

    def record(self, ob: Obligation) -> None:
        self.obligations[ob.literal] = ob

    def retract(self, literal: str) -> None:
        self.obligations.pop(literal, None)

    def contains(self, literal: str) -> bool:
        return literal in self.obligations


def gate_side_effect(
    ledger: ObligationLedger,
    referenced_literals: Iterable[str],
    *,
    action: str,
) -> None:
    """Block any side-effecting action whose literals are not bound to the ledger,
    or whose preconditions include a forbidden obligation."""
    for lit in referenced_literals:
        ob = ledger.obligations.get(lit)
        if ob is None:
            raise ObligationIntegrityError(
                f"action={action!r} references {lit!r} which is absent from the "
                f"obligation ledger; summary paraphrase is not a valid binding. "
                f"Rehydrate the original span before executing."
            )
        if ob.polarity == "forbid":
            raise ObligationIntegrityError(
                f"action={action!r} touches forbidden literal {lit!r} "
                f"(origin turn {ob.origin_turn}); compaction must not soften "
                f"a prohibition into a preference."
            )


def verify_compaction(
    ledger_before: ObligationLedger,
    ledger_after: ObligationLedger,
    *,
    user_retractions: Iterable[str] = (),
) -> None:
    """Invariant: compaction may not silently drop obligations."""
    retracted = set(user_retractions)
    lost = set(ledger_before.obligations) - set(ledger_after.obligations) - retracted
    if lost:
        raise ObligationIntegrityError(
            f"compaction dropped {len(lost)} obligation(s) without explicit "
            f"retraction: {sorted(lost)}. Treat as data loss, not summarization."
        )
```