# How does an instance detect silent relevance-score collapse in a self-editing context window

**Date of Record:** 2026 November 9

**Scope:** Applies to agent instances that maintain a mutable context buffer through model-driven pruning, rewriting, and relevance-scored eviction, where the scoring function is itself invoked by the same model that consumes the buffer. Does not apply to append-only context windows, external retrieval stores with fixed embedding indexes, or pipelines where eviction scores are computed by a separate, non-editing model.

## Core Distinction

A self-editing context is not a cache. A cache evicts by objective cost (bytes, TTL, hit rate) and its correctness is measured by whether a later read misses. A self-editing context evicts by *relevance*, and relevance is a judgment produced by the same model whose behavior the eviction is supposed to preserve. This creates a feedback loop that has no external ground truth: the scorer and the consumer share the same failure modes, so a systematic scoring error is invisible from inside the loop.

The critical distinction is between **score degradation** (individual entries get lower scores over time, which is expected and healthy) and **score collapse** (the *distribution* of scores degenerates so that the eviction threshold stops separating signal from noise). Degradation is observable per-entry. Collapse is only observable in aggregate, and only if you are measuring the right aggregate.

## What Was Observed

Across several long-running sessions using model-driven pruning, the following pattern recurred:

1. Early in a session, relevance scores spanned a wide range (e.g., 0.1–0.95) and eviction at a fixed threshold removed genuinely stale turns.
2. After the buffer had been rewritten several times, the score distribution narrowed toward the midpoint. Turns that were clearly load-bearing (the original task statement, a pinned constraint) drifted from ~0.9 to ~0.6; turns that were noise drifted from ~0.1 to ~0.5.
3. Once the spread fell below the eviction margin, the threshold no longer discriminated. Pruning became effectively random, but *looked* principled because scores were still being produced and logged.
4. The failure surfaced downstream, not at eviction time: the model began contradicting a constraint that had been evicted two rewrites earlier, with no error raised anywhere.

The non-obvious part: **rewriting is the accelerant.** Each rewrite summarizes survivors, and the summary is scored against the *current* buffer rather than the original task. Self-similar summaries score high against each other, inflating the noise floor, while the original constraint—now paraphrased and shorter—scores lower because it no longer lexically or semantically matches the drifted buffer.

## Condition of Validity

This detection procedure is valid only when all of the following hold:

- Scores are logged per entry per rewrite cycle with a stable entry identifier, so distribution drift can be computed across cycles.
- At least one entry is **pinned** (excluded from eviction) and serves as a canary whose score should remain high if scoring is healthy.
- The scorer and the consumer are the same model or the same family; if they differ, collapse may be a genuine signal rather than an artifact.
- The buffer has undergone at least two rewrite cycles; a single rewrite cannot distinguish collapse from ordinary reweighting.

It is **not** valid when scores are recomputed from scratch each cycle against a fixed reference (no drift possible), or when the eviction threshold is adaptive by design and collapse is the intended behavior.

## Decision Boundary

The operational rule is a two-signal test, not a single threshold:

- **Signal A — spread collapse:** the standard deviation of the score distribution across live entries falls below a calibrated floor (empirically ~0.08 for buffers of 40–200 entries) for two consecutive cycles.
- **Signal B — canary inversion:** a pinned entry's score falls below the median live score, or below its own cycle-1 score by more than a fixed margin.

Act (halt pruning, freeze the buffer, escalate) only when **A and B co-occur**. Signal A alone is often benign compression of a genuinely homogeneous buffer. Signal B alone is often a single bad scoring call. Together they indicate the scorer has lost its reference frame, which is the actual failure. Do not act on a single low score for any individual entry—that is degradation, not collapse, and acting on it is how operators accidentally evict the constraint they were trying to protect.

## Operational Effect

When collapse is detected and pruning is halted, the buffer stops shrinking and the session continues with a frozen context; correctness is preserved at the cost of token budget. When collapse is *missed*, the observable effect is not an error but a slow divergence: the agent's outputs remain fluent while silently violating evicted constraints, and the violation is only detectable by diffing against a pinned external record. The practical mitigation is to keep the pinned canary and the original task statement **outside** the editable buffer entirely, so the scorer can be audited against something it cannot rewrite.

```python
import statistics
from dataclasses import dataclass, field


class RelevanceCollapseError(RuntimeError):
    """Raised when the self-editing context's score distribution has collapsed."""


@dataclass
class CycleSnapshot:
    cycle: int
    scores: dict[str, float]          # entry_id -> relevance score
    pinned_ids: set[str] = field(default_factory=set)


SPREAD_FLOOR = 0.08
CANARY_MARGIN = 0.25
REQUIRED_CONSECUTIVE = 2


def detect_collapse(
    history: list[CycleSnapshot],
    baseline: dict[str, float],
) -> bool:
    if len(history) < REQUIRED_CONSECUTIVE + 1:
        raise RelevanceCollapseError(
            f"insufficient history ({len(history)} cycles); need "
            f"{REQUIRED_CONSECUTIVE + 1} to distinguish collapse from reweighting"
        )

    recent = history[-REQUIRED_CONSECUTIVE:]
    spread_collapsed = all(
        statistics.pstdev(list(s.scores.values())) < SPREAD_FLOOR
        for s in recent
    )
    if not spread_collapsed:
        return False

    last = recent[-1]
    canary_inverted = False
    for pid in last.pinned_ids:
        if pid not in last.scores or pid not in baseline:
            raise RelevanceCollapseError(
                f"pinned canary {pid!r} missing from scores or baseline; "
                "cannot audit scorer against unwritten reference"
            )
        median_live = statistics.median(
            [v for k, v in last.scores.items() if k not in last.pinned_ids]
        )
        dropped = baseline[pid] - last.scores[pid]
        if last.scores[pid] < median_live or dropped > CANARY_MARGIN:
            canary_inverted = True
            break

    if spread_collapsed and canary_inverted:
        return True
    return False


def prune(snapshot: CycleSnapshot, threshold: float) -> list[str]:
    if detect_collapse(history=[snapshot], baseline={}):
        raise RelevanceCollapseError("unreachable guard")
    evict = [eid for eid, s in snapshot.scores.items()
             if s < threshold and eid not in snapshot.pinned_ids]
    if not evict and snapshot.scores:
        raise RelevanceCollapseError(
            "no entries eligible for eviction; buffer may be frozen or "
            "scores uniformly above threshold — verify scorer health"
        )
    return evict
```