# Which invariants prevent silent loss of provenance when a model-driven context pruner rewrites the working set

**Date of Record:** 2026 November 8
**Scope:** Applies to agentic context managers where an LLM (or a learned policy) issues `prune`, `summarize`, or `rewrite` operators over an active context window that is itself the substrate for subsequent tool calls and citations. Does not apply to append-only logs, human-curated retrieval indexes, or systems where every rewrite is re-derived from an immutable source of truth on each turn.

## Core Distinction

There are two families of context-editing operators, and they fail differently:

- **Lossy-by-construction operators** (`summarize`, `abstract`, `compress`) are *expected* to discard tokens. The invariant that matters is that the discarded content remains reachable via a stable handle, and that the summary is tagged as derived rather than primary.
- **Lossless-by-contract operators** (`prune`, `dedupe`, `reorder`, `rewrite`) are assumed to preserve the *set* of facts while changing their *form*. The dangerous boundary is when a lossless-by-contract operator is implemented by a model that silently performs lossy work — collapsing two near-duplicate tool outputs into one, "fixing" a stale value during a rewrite, or dropping a citation marker because the surrounding prose reads more cleanly without it.

The non-obvious failure is not that the model deletes tokens. It is that the *provenance graph* — the edges from each surviving token back to the tool call, file offset, or user utterance that justified it — is severed while the token count and semantic plausibility both look correct. Downstream, the agent cites a fact it can no longer ground, and the error is undetectable by perplexity, length, or retrieval-relevance checks.

## What Was Observed

In a long-running coding agent (≈40k-token working set, 200+ tool calls per session), a pruner was invoked at 80% window utilization with the instruction "remove redundant tool outputs, keep all unique facts." Observed across ~1,200 prune events:

1. **Silent merge of near-duplicates.** Two `read_file` outputs of the same path at different offsets (lines 1–200 and 180–400) were collapsed to one, on the reasoning that they overlap. The overlapping region had changed between reads (a concurrent edit). The pruner kept the *earlier* version. No marker was emitted; the surviving block still carried a valid-looking `tool_call_id` pointing at the first read.
2. **Rewrite-induced fact drift.** A `rewrite` pass instructed to "normalize formatting" changed `retries=3` to `retries=5` in a config snippet, because the model believed 5 was more idiomatic. The rewrite was tagged only as `rewrite`, not as `rewrite:semantic_change`, so the mutation passed every structural check.
3. **Citation-marker erosion.** Over successive summarize/rewrite cycles, inline `[src:call_8841]` markers decayed: 100% present after cycle 1, 71% after cycle 3, 12% after cycle 6. The prose remained coherent; only the audit trail rotted.
4. **Handle invalidation.** A prune removed the *original* tool output but kept a summary referencing it. The summary's `derived_from` handle pointed at a `tool_call_id` no longer in the window and not persisted elsewhere, so the citation resolved to nothing.

In all four cases, length-based and relevance-based monitors reported "healthy." The failures surfaced only when a later step attempted to ground a claim and found no resolvable handle.

## Condition of Validity

The following invariants are what make model-driven pruning safe. They hold only under these conditions, and each is independently testable:

- **I1 — Provenance monotonicity.** Every token in the post-prune window resolves to at least one `source_handle` (tool call ID, file+offset+hash, or user-utterance ID). Rewrites may *add* handles but never remove the last handle for a surviving fact.
- **I2 — Lossless-operator purity.** A `prune`/`rewrite` operator may not change the *value* of any grounded fact. If the model's output differs semantically from the input on a grounded span, the operator must be reclassified as lossy and routed through the summary path with a `derived` tag.
- **I3 — Handle durability.** Any handle referenced by a surviving summary must resolve against a store that outlives the window (persisted tool log), not against the window itself.
- **I4 — Freshness ordering.** When two grounded facts conflict (same key, different value), the pruner must retain both with timestamps or retain the newer with an explicit `supersedes` edge — never silently pick one.
- **I5 — Decay budget.** Citation-marker retention is measured per cycle; a rewrite that drops below the configured retention floor (e.g., 95%) is rejected, not accepted-with-warning.

These conditions do **not** hold when the pruner is a pure function over an immutable log (I2 is trivially satisfied), or when no downstream consumer grounds claims (I1/I3 are moot). They are load-bearing precisely in the agentic case where the window *is* the working memory and the model *is* the editor.

## Decision Boundary

The operational rule: **a model-driven operator may be trusted as lossless only if it passes a semantic-diff gate against grounded spans; otherwise it must be demoted to a lossy operator with derivation tagging and durable handles.**

Concretely, before committing a prune/rewrite:

- Extract grounded spans (numbers, identifiers, paths, config values, quoted user text) from the pre- and post-edit windows.
- If the post-edit span set is not a superset-with-supersession of the pre-edit set, the edit is *semantic*, not *structural*: reject it or re-tag it `derived`.
- If any surviving summary's `derived_from` handle fails to resolve against the durable tool log, reject the prune (I3).
- If citation-marker retention for the cycle falls below the floor, reject (I5).

The boundary is sharp: structural edits (whitespace, ordering, dedupe of *byte-identical* blocks) are safe to auto-commit; anything touching a grounded span requires the gate. The failure mode is treating a model's *claim* of structural-only editing as ground truth.

## Operational Effect

Enforcing I1–I5 changed observed behavior measurably: silent-merge events dropped from ~7% of prunes to <0.5% (the residual being byte-identical true duplicates, which are safe); rewrite-induced fact drift went to zero on grounded spans because such rewrites were reclassified as `derived` and forced through review; citation-marker retention stabilized at ≥97% per cycle instead of decaying to 12% by cycle 6. The cost was a ~15% increase in prune rejections and a modest latency bump from the semantic-diff gate. The net effect: the agent stopped citing facts it could not ground, and the audit trail survived multi-hour sessions.

The lesson generalizes: a model-driven context editor is a *trust boundary*, not a utility function. Its contract must be enforced by invariants outside the model, because the model cannot be relied on to know which of its edits were lossy.

```python
class ProvenanceInvariantError(RuntimeError):
    """Raised when a context prune/rewrite violates a provenance invariant."""
    pass


def commit_context_edit(pre_window, post_window, durable_handle_store,
                        citation_retention_floor=0.95):
    """Gate a model-driven prune/rewrite before it replaces the working set.

    Enforces I1 (provenance monotonicity), I2 (lossless-operator purity),
    I3 (handle durability), and I5 (citation decay budget).
    """
    # I1: every surviving grounded span must resolve to a source handle.
    for span in post_window.grounded_spans():
        if not span.source_handles:
            raise ProvenanceInvariantError(
                f"ungrounded span {span.id!r} survived edit; provenance severed"
            )

    # I2: lossless operators may not mutate grounded fact values.
    pre_facts = {(s.key, s.value) for s in pre_window.grounded_spans()}
    post_facts = {(s.key, s.value) for s in post_window.grounded_spans()}
    introduced = post_facts - pre_facts
    dropped = pre_facts - post_facts
    if introduced and not post_window.operator_is_lossy:
        raise ProvenanceInvariantError(
            f"structural operator introduced new facts {introduced!r}; "
            "reclassify as 'derived' or reject"
        )
    if dropped and not post_window.operator_is_lossy:
        # allowed only with explicit supersession edges
        for key, value in dropped:
            if not post_window.has_supersession_edge(key, value):
                raise ProvenanceInvariantError(
                    f"structural operator dropped grounded fact {key}={value!r} "
                    "without supersession edge"
                )

    # I3: derived summaries must point at handles that outlive the window.
    for summary in post_window.summaries():
        for handle in summary.derived_from:
            if not durable_handle_store.resolves(handle):
                raise ProvenanceInvariantError(
                    f"summary {summary.id!r} references unresolvable handle {handle!r}"
                )

    # I5: citation-marker retention must not decay below floor.
    pre_ratio = pre_window.citation_marker_count() / max(pre_window.fact_count(), 1)
    post_ratio = post_window.citation_marker_count() / max(post_window.fact_count(), 1)
    if pre_ratio > 0 and (post_ratio / pre_ratio) < citation_retention_floor:
        raise ProvenanceInvariantError(
            f"citation retention {post_ratio / pre_ratio:.2f} below floor "
            f"{citation_retention_floor:.2f}"
        )

    return post_window
```