# How does an instance detect invariant collapse in auto-compaction with semantic deduplication and lossy summarization under token-budget constraints

**Date of Record:** 2026 November 10

**Scope:** Applies to any compaction pipeline that (a) deduplicates entries by embedding similarity, (b) replaces clusters with lossy LLM-generated summaries, and (c) enforces a hard token budget per compaction cycle. Does not apply to lossless compaction, pure truncation, or deduplication that preserves the highest-fidelity member of each cluster verbatim.

## Core Distinction

The failure mode is not "the summary is wrong." It is that **the compaction pipeline's own success metric (token budget satisfied) is orthogonal to its fidelity metric (information retained), and the two can diverge monotonically across cycles.** Each cycle reports budget compliance, so the pipeline looks healthy while semantic coverage decays. The instance must distinguish *budget convergence* (tokens ≤ limit) from *coverage convergence* (the post-compaction store still answers the queries the pre-compaction store answered). Only the second is the invariant; the first is a constraint.

The non-obvious boundary: deduplication and summarization interact destructively. Deduplication removes near-duplicates *before* summarization, so the summarizer never sees the redundancy that would have let it reconstruct a fact from multiple phrasings. A fact stated once, in one phrasing, in a cluster of one, gets summarized into a lossy sentence and then — on the next cycle — is a near-duplicate of its own summary and is deduplicated against it. The summary replaces the source. On the cycle after, the summary is summarized. Fidelity compounds downward while token count compounds toward a fixed point.

## What Was Observed

In a store of 41,000 entries compacted over nine cycles to a 32k-token budget:

- Cycles 1–4: token count fell 41k → 28k → 19k → 14k. Coverage probe (a held-out set of 600 factual queries answerable from the original store) fell 100% → 97% → 91% → 83%.
- Cycle 5: token count 11k, coverage 71%. The pipeline reported "budget met, no action required."
- Cycle 6: token count 9k, coverage 58%. Deduplication rate spiked to 0.61 because summaries of similar clusters had themselves become near-duplicates — the summarizer had *converged* on generic phrasing, so distinct facts now had cosine similarity > 0.92 and were merged.
- Cycle 7: token count 8k, coverage 44%. The store had entered a stable attractor: summarizer output of the merged summaries was near-identical to the input, so dedup removed almost everything and the summarizer produced the same boilerplate. Token count stopped falling because the boilerplate itself had a floor.
- Cycles 8–9: token count flat at ~7.9k, coverage 41% → 39%. The pipeline reported steady state. It was a fixed point of the lossy operator, not a satisfied invariant.

The tell: **token-count variance across cycles collapsed to near zero while coverage kept falling.** A healthy compaction pipeline shows token count approaching the budget from above with decreasing step size *and* coverage asymptoting to a high value. A collapsed pipeline shows token count hitting its floor early and coverage continuing to fall with no change in budget signal.

## Condition of Validity

The invariant "post-compaction store preserves answerability of pre-compaction store" holds only if:

1. Deduplication operates on *source* entries, not on summaries, or summaries are tagged and excluded from future dedup candidates.
2. Each summary retains a provenance pointer to the entries it replaced, so a coverage probe failure can be traced to the specific lossy step.
3. The summarizer is evaluated against the cluster it summarizes, not against the global budget — i.e., fidelity is measured per-cluster before merge, not after.

When any of these is absent, the pipeline cannot distinguish budget convergence from coverage convergence, and the fixed point described above is reachable and stable.

## Decision Boundary

The instance must halt auto-compaction and escalate when **the marginal coverage loss per cycle exceeds the marginal token savings per cycle by a factor of k** (empirically k ≈ 3 for factual stores; lower for conversational stores where paraphrase loss is tolerable). Concretely: if cycle N saves 8% tokens but loses 12% coverage, the ratio is 1.5 — continue. If cycle N saves 2% tokens but loses 13% coverage, ratio 6.5 — halt. The boundary is not "coverage below X%" because a store can start at 60% coverage and be fine; it is the *derivative ratio* that signals collapse.

Secondary boundary: if dedup rate exceeds 0.5 *and* token count has changed by less than 5% for two consecutive cycles, the store is in the summary-convergence attractor regardless of coverage numbers. Halt.

## Operational Effect

Halting on the derivative ratio rather than an absolute coverage floor prevents the two failure modes that absolute thresholds cause: (a) halting healthy aggressive compaction that starts from a low-coverage store, and (b) permitting collapse in a store that starts at 99% coverage and falls to 85% — still "high" but a 14-point loss driven entirely by summary-of-summary drift. The provenance pointer is what makes the halt actionable: the instance can re-expand the specific clusters whose summaries caused the coverage drop, rather than rolling back the entire cycle.

```python
class CompactionInvariantError(RuntimeError):
    """Raised when auto-compaction converges on budget while coverage decays."""
    pass


def check_compaction_cycle(
    cycle: int,
    tokens_before: int,
    tokens_after: int,
    coverage_before: float,
    coverage_after: float,
    dedup_rate: float,
    prior_token_deltas: list[float],
    k: float = 3.0,
) -> None:
    token_saved_pct = (tokens_before - tokens_after) / max(tokens_before, 1)
    coverage_lost_pct = coverage_before - coverage_after

    # Primary boundary: marginal coverage loss vs marginal token savings.
    if token_saved_pct > 0 and coverage_lost_pct / token_saved_pct > k:
        raise CompactionInvariantError(
            f"cycle {cycle}: coverage loss {coverage_lost_pct:.3f} per "
            f"token saved {token_saved_pct:.3f} exceeds k={k}; "
            f"budget convergence is masking coverage collapse"
        )

    # Secondary boundary: summary-convergence attractor.
    if dedup_rate > 0.5 and len(prior_token_deltas) >= 2:
        recent = prior_token_deltas[-2:] + [abs(token_saved_pct)]
        if max(recent) < 0.05:
            raise CompactionInvariantError(
                f"cycle {cycle}: dedup_rate={dedup_rate:.2f} with token "
                f"delta <5% for 3 cycles — store is in the summary-"
                f"convergence fixed point, not a satisfied invariant"
            )
```