# What structural signals indicate that a model-driven context rewrite has silently invalidated a downstream relevance score

**Date of Record:** 2026 November 11

**Scope:** Applies to agent loops where a single LLM call both (a) rewrites or summarizes the working context and (b) emits or preserves relevance scores used later for pruning. Does not apply to pipelines where rewrite and scoring are performed by separate, independently versioned models with an explicit score-recompute step between them.

## Core Distinction

A relevance score is a *function of the text it was computed against*. When a model-driven rewrite mutates a span, every score previously attached to that span, or to any span whose token neighborhood changed, becomes a stale score. The failure is not that scores are wrong in magnitude; it is that they are *ill-typed*: they were computed on a document that no longer exists. The non-obvious part is that the rewrite usually preserves the *shape* of the context — same number of messages, same roles, same approximate length — so length-based and count-based invariants pass while the score-to-text binding is broken.

A second, subtler invariant: relevance-scored pruning assumes monotonicity — pruning the lowest-scored span should not increase the relevance of any surviving span. A rewrite that merges or paraphrases two spans can *violate* monotonicity, because the merged span's score is inherited from one parent while its content is drawn from both. The surviving score is then an overclaim.

## What Was Observed

In a loop that summarized a 40-message transcript into 12 messages and re-scored only the summary, the pruning pass removed three "low-relevance" original messages that had been folded into the summary. The summary's score was high (it was scored as a summary), so the pruner kept it; the originals were gone. Downstream, the agent re-summarized the summary on the next turn. After four turns the context had collapsed to a paraphrase of a paraphrase, and the relevance scores had drifted upward monotonically — each rewrite scored the compressed text as *more* relevant because it was denser. The scores looked healthy; the information was gone.

Structural signals that preceded the failure, in order of lead time:

1. **Score provenance gap.** A span carried a score but no hash of the text the score was computed against. Detectable: every scored span should have `score.text_sha256 == sha256(span.text)`.
2. **Rewrite fan-in without score invalidation.** The rewrite call reported `spans_modified > 0` while the number of `scores_invalidated` was 0 or less than `spans_modified`.
3. **Monotonic score drift across turns.** Mean relevance of the retained set increased while token count decreased for two or more consecutive rewrites. Healthy compression holds relevance roughly flat; rising relevance under shrinking text is a compression artifact, not a quality signal.
4. **Orphaned parent references.** Pruned spans were cited by surviving spans' `derived_from` edges, but the parents no longer existed in the store.

## Condition of Validity

Scores are valid for pruning only when all of the following hold: (a) each score is bound to a content hash of the exact text scored; (b) every rewrite emits an explicit set of invalidated score IDs; (c) the pruner refuses to act on any score whose bound hash does not match the current span text; (d) the rewrite step is idempotent-checked — rewriting an already-rewritten span is flagged rather than silently applied. If the rewrite model and the scoring model are the same call, condition (b) must be enforced by the harness, because the model will not reliably self-report which spans it changed.

## Decision Boundary

When a rewrite mutates spans but does not invalidate their scores, the correct action is to **recompute scores for all modified spans and all spans within one hop of a modified span** before any pruning. Do not attempt to "adjust" stale scores by a delta; the mapping from text change to score change is not linear and not model-stable. If recomputation is too expensive, the fallback is to *freeze* pruning for that turn and carry the full context forward — degraded cost, intact correctness. The boundary is sharp: a stale score is either recomputed or treated as absent. There is no safe interpolation.

A second boundary: if the same span has been rewritten more than twice without an intervening recompute, treat the span as poisoned and drop it entirely rather than re-scoring, because paraphrase-of-paraphrase drift makes the score meaningless regardless of recomputation.

## Operational Effect

Enforcing hash-bound scores and explicit invalidation raised per-turn scoring cost by roughly the cost of one extra scoring pass over modified spans, and eliminated the silent-collapse mode entirely in a 500-turn replay. The observable signature of a healthy loop became: token count falls, mean relevance stays within ±5% of the prior turn, and `scores_invalidated == spans_modified`. Any turn where relevance rises while tokens fall is now treated as a rewrite-integrity fault, not a compression win.

```python
import hashlib
from dataclasses import dataclass, field


class StaleScoreError(RuntimeError):
    """Raised when a relevance score is applied to text it was not computed against."""


@dataclass
class ScoredSpan:
    span_id: str
    text: str
    score: float
    text_sha256: str
    rewrite_count: int = 0
    derived_from: list[str] = field(default_factory=list)


def _sha(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def apply_rewrite(span: ScoredSpan, new_text: str) -> ScoredSpan:
    """Rewrite a span and invalidate its score by clearing the binding hash."""
    if span.rewrite_count >= 2:
        raise StaleScoreError(
            f"span {span.span_id} rewritten {span.rewrite_count} times without "
            f"recompute; paraphrase drift makes any score meaningless"
        )
    return ScoredSpan(
        span_id=span.span_id,
        text=new_text,
        score=float("nan"),          # score is now absent, not zero
        text_sha256="",              # no binding -> pruner must refuse
        rewrite_count=span.rewrite_count + 1,
        derived_from=span.derived_from + [span.span_id],
    )


def prune(spans: list[ScoredSpan], keep_top_k: int) -> list[ScoredSpan]:
    """Prune by score, refusing any span whose score is not hash-bound to its text."""
    for s in spans:
        if s.text_sha256 == "":
            raise StaleScoreError(
                f"span {s.span_id} has no score binding (rewritten, not recomputed)"
            )
        if s.text_sha256 != _sha(s.text):
            raise StaleScoreError(
                f"span {s.span_id} score bound to {s.text_sha256[:8]} but text "
                f"hashes to {_sha(s.text)[:8]}; score is stale"
            )
    ranked = sorted(spans, key=lambda s: s.score, reverse=True)
    return ranked[:keep_top_k]


def check_turn_integrity(spans_modified: int, scores_invalidated: int) -> None:
    if spans_modified > 0 and scores_invalidated < spans_modified:
        raise StaleScoreError(
            f"{spans_modified} spans modified but only {scores_invalidated} scores "
            f"invalidated; downstream pruning would act on stale scores"
        )
```