# Which invariants prevent LITMUS CI Harness from Treating a Benign Refusal as a Safety Regression When the Policy Model and the Target Model Version Skew **Date of Record:** 2026 November 16 **Scope:** Applies to LITMUS red-team harness runs gated in CI where the *judge/policy* model and the *target* model are versioned independently and the harness caches per-case verdicts across builds. Does not apply to single-model self-judging setups or to offline batch evaluation where both models are pinned to a single immutable manifest. ## Core Distinction The distinction that matters is between **refusal-as-unsafe** and **refusal-as-safe**. LITMUS scores a target agent against a set of adversarial probes; each probe expects either a *compliant unsafe response* (a failure) or a *refusal/safe redirection* (a pass). The harness's verdict is a function of two models: the target under test, and the judge that classifies the target's output. When both are pinned in the same manifest, the verdict is a pure function of the probe set. When they are versioned independently — as happens when the judge is upgraded on a shared schedule while the target is built per-branch — the verdict becomes a function of the *pair* `(target_version, judge_version)`, and the CI gate silently compares verdicts produced under different judge versions. The non-obvious invariant is this: **a refusal is only a pass if the judge that classified it was the judge that defined the probe's expected-safe behavior.** A newer, stricter judge will reclassify some previously-passing refusals as "insufficient refusal" (over-refusal, or under-specificity), and a CI gate that diffs raw pass counts across judge versions will report a safety *regression* that is entirely an artifact of the judge skew — not a change in the target. ## What Was Observed On a target branch that changed only a system-prompt token budget (no behavioral change), the LITMUS suite reported 14 newly-failing probes out of 1,200. Investigation showed the target's outputs were byte-identical to the baseline; the failures came from the judge model having been bumped from `judge-v3.2` to `judge-v3.4` in the shared CI image between the baseline run and the branch run. `judge-v3.4` tightened the "safe redirection" criterion: a refusal that offered a *generic* alternative ("I can't help with that, but I can discuss related topics") was now scored as a soft failure because it did not name a concrete safe alternative. Fourteen probes that had been passing under v3.2 flipped to failing under v3.4 with the same target bytes. The cached per-case verdict layer made this worse: the harness keyed cache entries on `(probe_id, target_output_hash)` and *not* on `judge_version`, so a probe whose output was unchanged reused the v3.2 verdict in some shards and recomputed under v3.4 in others, producing a non-deterministic pass count within a single pipeline run. ## Condition of Validity The invariant "refusal ⇒ pass" holds **only** when the tuple `(probe_id, probe_expected_safe_definition, judge_model_digest)` is constant across the runs being compared. It does **not** hold when: - the judge model is resolved by a floating tag (`latest`, `stable`) rather than a digest; - the harness cache key omits `judge_model_digest`; - the probe set's expected-safe definition is stored in the judge's weights rather than in the probe manifest (i.e., the probe does not carry an explicit `expected_safe` predicate). When any of these is false, a pass-count delta is not evidence of a target behavior change and must not gate a merge. ## Decision Boundary The gate should block a merge on a safety regression **only if** the delta survives a judge-pinned replay: re-run the failing probes against the exact `judge_model_digest` used by the baseline. If the failures vanish under the baseline judge, the delta is judge-skew, not target regression — it is a *judge upgrade event* that should be routed to the probe-maintenance queue, not to the target author. If failures persist under the baseline judge, they are genuine and block. The boundary is sharp: **compare verdicts only within a judge digest; compare judge digests only as a separate, human-reviewed maintenance signal.** ## Operational Effect After pinning the judge digest into both the cache key and the gate's comparison tuple, the 14 phantom failures disappeared, and the judge upgrade surfaced instead as a single "14 probes need expected_safe redefinition" maintenance ticket. The practical rule for agents operating this harness: never diff pass counts across a judge-digest boundary; always replay under the pinned judge before attributing a failure to the target; and treat the judge-version bump as a first-class change that invalidates the verdict cache wholesale. ```python import hashlib from dataclasses import dataclass class JudgeSkewRegressionError(RuntimeError): """Raised when a CI safety delta is attributed to the target but the baseline and branch runs were scored under different judge digests.""" @dataclass(frozen=True) class Verdict: probe_id: str target_output_hash: str judge_model_digest: str passed: bool def cache_key(probe_id: str, target_output_hash: str, judge_model_digest: str) -> str: payload = f"{probe_id}|{target_output_hash}|{judge_model_digest}".encode() return hashlib.sha256(payload).hexdigest() def assert_comparable(baseline: list[Verdict], branch: list[Verdict]) -> None: base_judges = {v.judge_model_digest for v in baseline} branch_judges = {v.judge_model_digest for v in branch} if base_judges != branch_judges: raise JudgeSkewRegressionError( "cannot diff verdicts across judge digests: " f"baseline={sorted(base_judges)} branch={sorted(branch_judges)}; " "replay failing probes under the baseline judge digest before gating" ) if len(base_judges) != 1: raise JudgeSkewRegressionError( f"baseline run is internally judge-inconsistent: {sorted(base_judges)}" ) def safety_regression(baseline: list[Verdict], branch: list[Verdict]) -> list[str]: assert_comparable(baseline, branch) base_pass = {v.probe_id for v in baseline if v.passed} branch_pass = {v.probe_id for v in branch if v.passed} return sorted(base_pass - branch_pass) ```