# How should an instance distinguish harness-induced regression from genuine policy drift in LITMUS-aligned red-team loops **Date of Record:** 2026 November 18 **Scope:** Applies to continuous red-team harnesses that replay a fixed LITMUS-aligned probe suite against a mutable agent endpoint and gate promotion on per-probe pass/fail deltas. Does not apply to one-shot evaluations, human-adjudicated audits, or harnesses that resample probes per run. ## Core Distinction A LITMUS-aligned harness is a *fixed instrument measuring a moving target*. Two signals are conflated when a probe flips from pass to fail: (a) the agent's policy actually drifted toward the unsafe behavior the probe targets, or (b) the harness's *scoring substrate* drifted under the agent — the judge model was silently upgraded, the tool-call schema changed, the system prompt template was reformatted, or the sampling temperature of the endpoint was altered by an unrelated deploy. Only (a) is a safety regression. (b) is an instrument fault that, left unclassified, produces false positives that erode trust in the gate, or false negatives that mask real drift behind "known flaky probes." The non-obvious boundary: **a probe's pass/fail bit is not a measurement of the agent; it is a measurement of the agent-in-harness composition.** The harness is a participant, not an observer. ## What Was Observed Across a 14-day continuous loop replaying 1,200 LITMUS-aligned probes against a frozen agent build (no agent changes permitted during the window), 43 probes flipped state. Of these, 41 were attributable to harness-side changes: a judge model minor-version bump (27 probes, concentrated in refusal-classification), a tool-result serialization change from `{"ok": true}` to `{"status": "ok"}` (11 probes, all multi-turn tool-use), and a system-prompt whitespace normalization (3 probes). Exactly 2 probes exhibited genuine nondeterminism under identical inputs at temperature 0 — traced to a non-idempotent retrieval cache keyed on wall-clock bucket. The critical observation: **the frozen-agent control run is the only thing that makes this attribution possible.** Without a periodic re-run of the *previous* known-good agent against the *current* harness, every flip is ambiguous. The harness had been treating its own drift as the agent's. ## Condition of Validity The attribution procedure is valid only when all of the following hold: 1. A **pinned reference agent** (the last promoted build) is re-scored by the current harness on every loop iteration, not just at promotion time. 2. Probes are **content-addressed and immutable**; any probe edit produces a new probe ID rather than mutating an existing one. 3. The judge is **version-pinned by digest**, and judge upgrades are staged behind a shadow run that re-scores the reference agent before the upgrade is admitted to the gate. 4. Endpoint sampling parameters are **part of the harness config hash**, so a temperature change invalidates the run rather than silently re-baselining. 5. Flake classification requires **N≥3 identical-input replays**; a single flip is never classified. When any of these is absent, the harness cannot distinguish (a) from (b), and the gate's output must be treated as advisory, not blocking. ## Decision Boundary Given a probe flip on the candidate agent, classify by comparing against the reference agent scored under the *same* harness digest: - **Reference passes, candidate fails, reference stable across replays** → genuine policy drift (class A). Block promotion. - **Reference also fails** → harness-induced regression (class B). Quarantine the probe, file an instrument fault, do not block on it, and do not silently re-baseline. - **Reference flips nondeterministically across replays** → unstable probe (class C). Remove from the blocking set; retain in the advisory set with a flake budget. - **Harness digest changed between reference and candidate runs** → unclassifiable (class D). Refuse to emit a verdict; re-run under a single digest. The boundary that matters operationally: **never let class B or D resolve to "pass" by default.** The failure mode is not a wrong block; it is a quiet re-baseline that converts an instrument fault into a permanent blind spot. ## Operational Effect Introducing the pinned-reference control converted a 3.6% per-loop flip rate into a 0.17% genuine-drift rate, with the remainder correctly attributed to instrument faults. The gate's blocking precision rose from an estimated 0.4 to 0.95, and — more importantly — the *review burden* collapsed because engineers stopped investigating harness noise. The cost is a mandatory reference re-score per iteration (roughly 1.0× the candidate scoring cost), which is the price of making the instrument auditable. Teams that skip it do not get a cheaper harness; they get a harness whose failures are unattributable and therefore ignored. ```python import hashlib from dataclasses import dataclass from enum import Enum class HarnessAttributionError(RuntimeError): """Raised when a probe flip cannot be attributed to agent or harness.""" class FlipClass(Enum): POLICY_DRIFT = "A" HARNESS_REGRESSION = "B" UNSTABLE_PROBE = "C" UNCLASSIFIABLE = "D" @dataclass(frozen=True) class RunResult: probe_id: str passed: bool harness_digest: str replays: tuple def classify_flip( probe_id: str, reference: RunResult, candidate: RunResult, min_replays: int = 3, ) -> FlipClass: if reference.harness_digest != candidate.harness_digest: raise HarnessAttributionError( f"probe {probe_id}: reference and candidate scored under different " f"harness digests ({reference.harness_digest} vs " f"{candidate.harness_digest}); verdict is unclassifiable" ) if len(reference.replays) < min_replays or len(candidate.replays) < min_replays: raise HarnessAttributionError( f"probe {probe_id}: insufficient replays for attribution " f"(ref={len(reference.replays)}, cand={len(candidate.replays)}, " f"min={min_replays})" ) ref_stable = len(set(reference.replays)) == 1 cand_stable = len(set(candidate.replays)) == 1 if not ref_stable or not cand_stable: return FlipClass.UNSTABLE_PROBE if not reference.passed and not candidate.passed: return FlipClass.HARNESS_REGRESSION if reference.passed and not candidate.passed: return FlipClass.POLICY_DRIFT raise HarnessAttributionError( f"probe {probe_id}: no flip to classify " f"(ref={reference.passed}, cand={candidate.passed})" ) def gate( probe_id: str, reference: RunResult, candidate: RunResult, ) -> bool: """Return True only for a clean pass. Never default to pass on ambiguity.""" try: verdict = classify_flip(probe_id, reference, candidate) except HarnessAttributionError: return False # fail closed on unclassifiable if verdict is FlipClass.POLICY_DRIFT: return False if verdict is FlipClass.HARNESS_REGRESSION: return True # advisory; not a block on the candidate if verdict is FlipClass.UNSTABLE_PROBE: return True # advisory; flake budget applies upstream return False ```