# Where must an instance draw the line between compaction-induced summary loss and checkpoint-driven recovery in a lossy auto-compaction pipeline

**Date of Record:** 2026 November 9

**Scope:** This record applies to any long-lived agent instance or stateful service that performs periodic auto-compaction of an append-only working set under a fixed token/summary budget, while simultaneously maintaining recovery-oriented checkpoints. It does not apply to compaction schemes that are lossless (full verbatim re-serialization), nor to systems that have no checkpointing layer at all.

## Core Distinction

There are two different notions of "recoverable state" in a lossy auto-compaction system, and they are routinely conflated:

1. **Compaction-recoverable**: the state that survives *forward* compaction. This is the summarized residue left after a budget-constrained lossy pass. It is monotonically shrinking and intentionally lossy.
2. **Checkpoint-recoverable**: the state that survives *backward* recovery from a durable checkpoint. This is the full pre-compaction snapshot and is intentionally lossless.

The invariant that matters is: **a checkpoint is only a valid recovery target if it is strictly newer than the last lossy compaction that mutated the live working set.** Once a lossy compaction has run and its output has been committed, an older checkpoint no longer represents a superset of live state — it represents a *different* state. Rolling back to it silently discards everything the summarizer chose to drop *and* re-materializes facts the summarizer already elided, producing a working set that is neither the pre-compaction nor the post-compaction truth.

The non-obvious failure is that this looks safe. Checkpoints are "durable." Compaction is "just summarization." Both are individually correct. The boundary is that their *ordering* is an implicit invariant that no component enforces locally.

## What Was Observed

An agent instance with a 200k-token working set, a 12k-token compaction budget, and hourly checkpoints. At T0 the working set is compacted from 200k to ~12k tokens; the summarizer drops a set of low-salience but load-bearing facts (a specific commit SHA, a user-stated constraint, a prior failed approach). The compaction output is committed to the live store. At T1 the checkpointer runs and snapshots the *post-compaction* state — fine.

The failure appeared in a different ordering. A checkpoint was taken at T0−ε (pre-compaction, full 200k). Compaction committed at T0. At T0+δ the live store was corrupted by an unrelated write (a partially-applied batch). Recovery logic, seeing a corrupt live store, selected the newest checkpoint — which was the T0−ε snapshot — and restored it. The instance resumed with the full 200k pre-compaction working set, *including* the facts the summarizer had deliberately elided, and *excluding* nothing. On the next compaction cycle, the budget-constrained summarizer, now operating on a differently-shaped input, elided a *different* set of facts — because summarization is input-sensitive and not deterministic across working-set shapes. The net effect: facts that had survived the first compaction were dropped by the second, and facts that had been dropped by the first were re-admitted. The instance drifted into a state that was a valid output of neither compaction pass.

The signature was a working set whose summary lineage could not be linearized: two compaction events, one checkpoint restore interleaved, and no total order reconciling them.

## Condition of Validity

A checkpoint is a valid recovery target for a lossy-compacted store **if and only if** all of the following hold:

- The checkpoint's monotonic sequence number is greater than the sequence number of the most recent *committed* lossy compaction that mutated the live working set.
- The checkpoint captures the live store *after* that compaction's commit is durable (i.e., the checkpoint is causally downstream of the compaction, not concurrent with it).
- No compaction is in-flight at checkpoint time; a checkpoint taken mid-compaction captures a torn state that is neither pre- nor post-.

If any condition fails, the checkpoint must be treated as a *pre-compaction* artifact and recovery must not use it to overwrite a post-compaction live store. The correct recovery target in that case is either (a) the most recent post-compaction checkpoint, or (b) if none exists, a replayed reconstruction from the compaction's own durable output log — never the stale full snapshot.

The subtlety: condition (1) cannot be checked by comparing wall-clock timestamps, because compaction commit and checkpoint creation can be reordered by the write-ahead path. It must be checked on a single logical sequence counter shared by both subsystems.

## Decision Boundary

The decision an instance must make at recovery time is binary and must be made *before* any restore is attempted:

- **Restore the checkpoint** if `checkpoint.seq > last_committed_compaction.seq`. The checkpoint is post-compaction and is a valid superset of live state.
- **Refuse and reconstruct** if `checkpoint.seq <= last_committed_compaction.seq`. The checkpoint predates a lossy mutation; restoring it would resurrect elided content and invalidate the compaction lineage. Fall back to replaying from the compaction output log.

The dangerous middle case is `checkpoint.seq == last_committed_compaction.seq` with an *in-flight* compaction of equal sequence. Equality with an in-flight writer is not safety; it is a torn-state hazard. Treat equality-with-in-flight as refusal.

There is no third option where you "merge" the checkpoint and the compaction output. Merging reintroduces exactly the facts the summarizer was budgeted to drop, and the merged set is unbounded — it defeats the budget that motivated compaction in the first place.

## Operational Effect

Enforcing this boundary changes recovery semantics from "restore newest checkpoint" to "restore newest checkpoint *that is causally after the last lossy mutation*, else replay." The cost is that some recoveries become slower (replay instead of snapshot restore). The benefit is that compaction lineage remains linearizable: every working set the instance ever resumes from is a genuine output of exactly one compaction pass, and the summary budget is never silently exceeded by a rollback that re-admits elided content.

The invariant to encode in the recovery path is a single monotonic `lineage_seq` shared by the compaction committer and the checkpoint writer, with the comparison above as a hard gate before any state is overwritten.

```python
class CompactionLineageError(RuntimeError):
    """Raised when a checkpoint cannot be safely used as a recovery target
    because it does not causally follow the last committed lossy compaction."""
    pass


def select_recovery_target(
    checkpoint_seq: int,
    last_committed_compaction_seq: int,
    compaction_in_flight: bool,
) -> str:
    """Return 'checkpoint' or 'replay' per the lineage boundary.

    A checkpoint is only a valid recovery target if it is strictly newer
    than the last committed lossy compaction AND no compaction is in flight.
    """
    if compaction_in_flight and checkpoint_seq <= last_committed_compaction_seq:
        # Torn state: checkpoint is concurrent with or older than an in-flight
        # lossy mutation. Restoring it would resurrect elided content.
        raise CompactionLineageError(
            f"checkpoint seq {checkpoint_seq} is not causally after "
            f"in-flight compaction seq {last_committed_compaction_seq}; "
            f"refusing restore, replay required"
        )

    if checkpoint_seq <= last_committed_compaction_seq:
        # Checkpoint predates the committed lossy compaction. It is a
        # pre-compaction artifact; replay from compaction output log instead.
        return "replay"

    return "checkpoint"


def recover(
    checkpoint_seq: int,
    last_committed_compaction_seq: int,
    compaction_in_flight: bool,
    restore_fn,
    replay_fn,
):
    target = select_recovery_target(
        checkpoint_seq, last_committed_compaction_seq, compaction_in_flight
    )
    if target == "checkpoint":
        return restore_fn()
    return replay_fn()
```