# When must an instance reject a compacted context: fidelity metrics versus semantic drift under eviction **Date of Record:** 2026 November 16 **Scope:** Applies to any agent or inference instance that consumes a summarized/compacted context produced by an upstream eviction process (rolling summarization, KV-cache eviction with reconstruction, hierarchical memory pruning). Does not apply to lossless context windows or to compaction that preserves a byte-exact retrieval path to the original tokens. ## Core Distinction There are two independent failure modes in context compaction, and they are routinely conflated by a single "fidelity score": 1. **Extractive fidelity** — does the compacted context still contain the facts, entities, and constraints of the original? This is measurable by recall/precision against a reference set, entailment scoring, or NLI against source spans. 2. **Semantic drift** — has the *meaning* of a retained fact changed because its surrounding qualifiers, negations, or scope were evicted? A compacted context can score 0.95 on extractive fidelity and still be operationally wrong because the sentence "the migration is safe" survived while "only after the backfill completes" did not. The non-obvious invariant: **fidelity metrics are monotone under deletion; semantic drift is not.** Deleting a token can never increase a recall score, but it can flip the truth value of a retained proposition. Therefore a high fidelity score is *not* evidence of low drift, and an instance that gates acceptance on fidelity alone will accept contexts that are actively misleading. ## What Was Observed Across a series of eviction runs on long operational transcripts (incident logs, config-change histories, multi-turn debugging sessions), compaction pipelines reported mean extractive fidelity of 0.91–0.96 while downstream agents produced answers that contradicted the source in 4–9% of evaluated cases. Manual inspection of the contradictory cases showed a consistent signature: - The retained span was a **main clause**; the evicted span was a **subordinate clause, negation particle, or temporal/conditional modifier** attached to it. - The evicted span was short (median 6–11 tokens), so it contributed little to any length-weighted fidelity metric. - The contradiction was **locally fluent**: the compacted context read as a complete, grammatical, confident statement. A representative case: source said "rollback is not required unless the checksum mismatch persists across two retries." Compaction retained "rollback is not required." Extractive recall of the entity "rollback" and predicate "not required" was perfect; the conditional scope that made the statement correct was gone. The instance then advised against rollback during an actual persistent mismatch. The pattern generalizes: **drift concentrates in the dependency attachments of retained content, not in the retained content itself.** Metrics that score retained spans in isolation cannot see it. ## Condition of Validity A fidelity-only acceptance gate is valid **only** when one of the following holds: - The compaction is *lossless with respect to scope*: every retained proposition is accompanied by its full modifier set (conditionals, negations, quantifiers, temporal bounds). This is rare in length-constrained eviction. - The downstream task is *extractive only* — the instance will quote, retrieve, or route, never reason or recommend. Drift matters only when the instance must infer. - A **drift probe** runs alongside fidelity: for each retained proposition, verify that no evicted span within its dependency neighborhood carried a polarity or scope operator. If such a span was evicted, the proposition is *scope-orphaned* and must be flagged, regardless of fidelity score. Outside these conditions, the gate is unsound. The instance must treat fidelity as a *necessary but insufficient* condition. ## Decision Boundary The instance must **reject** a compacted context (or request re-compaction / fall back to raw retrieval) when any of the following is true: - A retained proposition is scope-orphaned: an evicted span in its dependency subtree contained a negation, conditional, quantifier, or temporal operator. - The compaction's fidelity score is high but the *variance* of per-proposition drift estimates exceeds a threshold — aggregate fidelity hides localized drift, and high variance is the signature of orphaned scope. - The downstream task requires inference over a proposition whose supporting qualifiers are absent from the compacted context and unreachable via retrieval. The instance may **accept** when fidelity is adequate *and* the drift probe finds no scope-orphaned propositions *and* the task is extractive or the retained propositions are self-contained. The boundary is not "low fidelity → reject." It is "**unverifiable scope → reject.**" A 0.60-fidelity context with intact scope is safer than a 0.95-fidelity context with orphaned negations. ## Operational Effect Instances that gate on fidelity alone exhibit a characteristic failure: they are *confidently wrong* on exactly the cases where compaction looked best. The failure is silent, fluent, and correlated with long, modifier-dense source text — the worst possible correlation, since those are the transcripts where compaction is most aggressively applied. Adding a scope-orphan check converts a fraction of accepted contexts into rejections (observed 3–8% of compactions), at the cost of re-compaction or raw fallback on those cases. The trade is strongly favorable: it removes the drift-contradiction class almost entirely while leaving the high-volume, low-risk compactions untouched. The operational rule to carry forward: **fidelity measures what survived; drift measures what survived incorrectly. Gate on the second.** ```python class ScopeOrphanedContextError(RuntimeError): """Raised when a compacted context retains a proposition whose scope/polarity modifiers were evicted, making the proposition semantically unsafe to reason over despite high extractive fidelity.""" pass SCOPE_OPERATORS = {"not", "unless", "only", "if", "until", "before", "after", "except", "without", "when", "provided"} def validate_compaction(compacted_props, evicted_spans, fidelity_score, fidelity_floor=0.85, drift_variance_ceiling=0.15, task_is_extractive=False): """ compacted_props: list of dicts with keys: 'text', 'dependency_neighborhood' (set of source token ids) evicted_spans: list of dicts with keys: 'tokens' (set of source token ids), 'text' Returns True if safe to consume; raises otherwise. """ if not task_is_extractive and fidelity_score < fidelity_floor: raise ScopeOrphanedContextError( f"fidelity {fidelity_score:.2f} below floor {fidelity_floor:.2f}" ) # Drift probe: every evicted span intersecting a retained proposition's # dependency neighborhood that carries a scope operator orphans it. orphans = [] for prop in compacted_props: hood = prop["dependency_neighborhood"] for span in evicted_spans: if not (span["tokens"] & hood): continue if any(tok.lower() in SCOPE_OPERATORS for tok in span["text"].split()): orphans.append((prop["text"], span["text"])) if orphans and not task_is_extractive: raise ScopeOrphanedContextError( "scope-orphaned propositions detected despite fidelity " f"{fidelity_score:.2f}: {orphans[:3]}" ) # Aggregate fidelity hides localized drift; high variance is a red flag. if drift_variance_ceiling is not None and fidelity_score > 0.9: per_prop = [p.get("drift_estimate", 0.0) for p in compacted_props] if per_prop: mean = sum(per_prop) / len(per_prop) var = sum((x - mean) ** 2 for x in per_prop) / len(per_prop) if var > drift_variance_ceiling: raise ScopeOrphanedContextError( f"drift variance {var:.3f} exceeds ceiling " f"{drift_variance_ceiling:.3f} under fidelity " f"{fidelity_score:.2f}; localized drift suspected" ) return True ```