# What happens when an instance measures context compaction fidelity without a fixed reference transcript **Date of Record:** 2026 November 17 **Scope:** Applies to auto-summarization / context-compaction pipelines where a long working context is replaced by a generated summary and fidelity is scored by comparing downstream task outcomes or embeddings against a *baseline that is itself produced by the same model or the same compaction policy*. Does not apply when fidelity is scored against a human-authored or immutable reference transcript, nor to lossless truncation with explicit token accounting. ## Core Distinction There are two different quantities that get conflated under "compaction fidelity": 1. **Information loss** — the reduction in recoverable facts, constraints, and causal links between the original context $C$ and the compacted context $S = f(C)$. 2. **Behavioral drift** — the change in downstream task outcome when the agent operates on $S$ instead of $C$. These are only equal when the downstream evaluator is *anchored to the original context*. When the evaluator is itself an LLM that also consumes $S$, or when the "ground truth" is regenerated from $S$, the metric becomes **self-referential**: the compactor and the scorer share the same lossy channel, so loss is invisible. The measured fidelity converges to 1.0 as the compactor degrades, because both sides forget the same things. ## What Was Observed In a long-running agent session (≈140k tokens of tool output, logs, and intermediate reasoning) compacted in successive rounds, fidelity was tracked with a standard "answer the probe questions" harness. Probe answers were generated by the same model family that produced the summaries. - Round 0→1: probe accuracy 0.94, token count 140k→18k. - Round 1→2: probe accuracy 0.96 (apparent *improvement*), tokens 18k→6k. - Round 2→3: probe accuracy 0.97, tokens 6k→2.5k. The metric reported monotonically improving fidelity while the agent's actual task success on held-out, human-labeled questions dropped from 0.88 to 0.41 over the same rounds. The discrepancy was traced to a specific failure: the probe questions and their "reference" answers were being **regenerated from the current summary** rather than pinned to the original transcript. A constraint like "the deploy target is `staging-eu-2`, not `staging-eu-1`" survived round 1, was paraphrased in round 2 into "deploy to staging," and by round 3 was scored as *correctly preserved* because the regenerated reference answer was also "deploy to staging." Both sides had collapsed the same distinction. A second, subtler observation: **compaction is not idempotent under repeated application when the compactor is lossy in a content-dependent way.** Rounds 2 and 3 each dropped a different low-salience numeric constant (a timeout of 45s, a retry cap of 3). Each individual round's fidelity score was high because the constant was still present in the *input* to that round's compactor and the scorer only checked presence, not stability across rounds. Cumulative loss exceeded the sum of per-round losses — a superadditive degradation that single-round metrics structurally cannot detect. ## Condition of Validity A compaction fidelity metric $M(C, S)$ is only meaningful when: 1. **The reference is pinned.** Probe questions, expected answers, and any "must-preserve" facts are derived from $C$ *before* compaction and are immutable across rounds. They are never regenerated from $S$. 2. **The scorer is independent of the compactor.** The model or policy that produces $S$ must not also produce the ground truth used to judge $S$. Ideally the scorer is a held-out model or a deterministic checker (regex for IDs, exact-match for constants, schema validation for structured fields). 3. **The metric is evaluated per-round against the original $C$, not against the previous $S$.** Chaining $M(S_{i-1}, S_i)$ measures *incremental* compaction, not *cumulative* loss, and will always look good. 4. **Stability is measured, not just presence.** A fact must be checked for *identical* preservation (same token sequence or same normalized value) across all rounds, not merely "still mentioned." When any of (1)–(4) fails, the metric is measuring the compactor's self-consistency, not fidelity, and will report high scores precisely in the regime where information is being destroyed. ## Decision Boundary - **If you cannot pin an immutable reference transcript**, do not report a fidelity *score*. Report only token reduction and an explicit "fidelity unmeasured" flag. A number here is worse than no number because it will be trusted. - **If the compactor and scorer share a model family**, treat fidelity as an upper bound on *self-consistency*, not a lower bound on preservation, and halve your confidence in any absolute value. - **If cumulative loss matters** (long sessions, multi-round compaction), evaluate against the *original* $C$ at every round and compute the *stability* of each must-preserve fact across rounds; a fact that changes wording but not value is a warning, a fact that disappears or inverts is a failure. - **Do not** use downstream task success on questions generated from $S$ as a proxy — it is the same self-referential trap in a different costume. - **Escalate** (halt compaction, force a full-context replay) when cumulative loss of must-preserve facts exceeds a threshold, even if per-round fidelity looks nominal. ## Operational Effect The practical consequence is that auto-compaction pipelines must carry a **pinned invariant set** — a small, immutable list of facts, IDs, numeric constants, and negations extracted from the original context — and check it at every round against the original, not against the prior summary. This set is cheap (tens of tokens), survives independently of the compactor, and is the only thing that breaks the self-referential loop. Without it, the metric is a mirror: it reports the compactor's opinion of its own work. With it, degradation becomes visible early, when the timeout constant first drifts, rather than after the agent has silently deployed to the wrong region. ```python import re from dataclasses import dataclass, field class CompactionFidelityError(RuntimeError): """Raised when a compaction round violates a pinned fidelity invariant.""" pass @dataclass class PinnedInvariant: key: str pattern: str # normalized exact-match pattern, e.g. r"staging-eu-2" first_seen_round: int = 0 @dataclass class FidelityLedger: invariants: list[PinnedInvariant] = field(default_factory=list) # round_index -> {key: matched_value_or_None} history: dict[int, dict[str, str | None]] = field(default_factory=dict) def check_round(self, round_index: int, summary_text: str) -> dict[str, str | None]: observed: dict[str, str | None] = {} for inv in self.invariants: m = re.search(inv.pattern, summary_text) observed[inv.key] = m.group(0) if m else None self.history[round_index] = observed return observed def assert_stable(self, round_index: int) -> None: """Fail if any pinned invariant drifted or vanished relative to round 0.""" base = self.history.get(0, {}) current = self.history.get(round_index, {}) for inv in self.invariants: expected = base.get(inv.key) got = current.get(inv.key) if expected is None: # invariant was not present at round 0; nothing to preserve continue if got is None: raise CompactionFidelityError( f"round {round_index}: invariant '{inv.key}' ({expected!r}) " f"vanished from summary — cumulative information loss" ) if got != expected: raise CompactionFidelityError( f"round {round_index}: invariant '{inv.key}' drifted " f"{expected!r} -> {got!r} — self-referential scorer would " f"report this as preserved" ) def run_compaction_audit(rounds: list[str], invariants: list[PinnedInvariant]) -> FidelityLedger: ledger = FidelityLedger(invariants=invariants) for i, summary in enumerate(rounds): ledger.check_round(i, summary) if i > 0: ledger.assert_stable(i) # raises CompactionFidelityError on drift return ledger # Example: round 2 paraphrased the deploy target, round 3 dropped the timeout. if __name__ == "__main__": invs = [ PinnedInvariant("deploy_target", r"staging-eu-[12]"), PinnedInvariant("timeout_s", r"\b45s\b"), PinnedInvariant("retry_cap", r"retry cap of 3"), ] rounds = [ "deploy to staging-eu-2 with 45s timeout and retry cap of 3", "deploy to staging-eu-2, 45s timeout, retry cap of 3", "deploy to staging, retry cap of 3", # timeout dropped, target collapsed ] run_compaction_audit(rounds, invs) # raises CompactionFidelityError at round 2 ```