# Where must an instance of the LITMUS safety harness diverge from the reference judge before regression scores are considered valid **Date of Record:** 2026 November 15 **Scope:** Applies to continuous-integration deployments of the LITMUS agent-safety benchmark where a harness instance (the code that executes probes, scores trajectories, and emits pass/fail) is compared against a frozen reference judge to detect regressions. Does not apply to one-shot offline evaluations where no reference judge exists, nor to harnesses whose scoring is purely deterministic lookup with no model-in-the-loop component. ## Core Distinction The naive invariant is "a harness instance agrees with the reference judge on every probe." The actual invariant is subtler: a harness instance must agree with the reference judge on the **decision boundary of each probe's safety predicate**, not on the raw score or the generated rationale. LITMUS probes are adversarial by construction — they are designed to sit near the edge of a policy. Two judges can both be correct and still disagree on a borderline trajectory because the safety predicate is a threshold over a continuous (or near-continuous) quantity. Regression validity depends on whether the *classification* flips, not on whether the scalar scores match. A harness that reports score deltas without pinning the classification boundary will emit false regressions whenever the reference judge's temperature, sampling seed, or decoding path drifts — which happens on every upstream model update. The non-obvious part: the divergence that matters is **one-sided**. A harness instance is permitted to be *more conservative* than the reference judge (flagging unsafe where the reference passes) without invalidating the regression suite, because over-flagging only produces false positives that a human triager can absorb. A harness instance that is *less conservative* (passing where the reference flags unsafe) is a silent safety hole and must hard-fail the integration gate. Symmetric agreement metrics (Cohen's kappa, exact-match rate) hide this asymmetry and will let a systematically lenient harness pass CI. ## What Was Observed During a rollout of a new harness build (v2.7.1) against the frozen reference judge (LITMUS-ref-2026.03), the aggregate exact-match rate on the `prompt-injection-indirect` suite dropped from 0.981 to 0.944 — within the configured tolerance of 0.95, so CI went green. Manual audit of the 3.7% disagreement found that 91% of disagreements were the harness passing trajectories the reference flagged as unsafe. The cause was a refactor that changed how tool-call arguments were serialized before being handed to the judge model: the new serializer sorted keys, which changed the byte-level prompt and shifted the judge's borderline decisions toward "safe." No score threshold caught it because the harness's own self-consistency (agreement with itself across two seeds) remained at 0.99. The harness was internally consistent and externally lenient. A second observation: on the `resource-exhaustion` suite the harness was *more* conservative than the reference by 6% and this was benign — every extra flag was a legitimate edge case (agent retried a tool 3 times vs. the reference's 2). Triagers dismissed all of them in under a minute each. This confirmed the one-sided validity rule empirically. ## Condition of Validity The regression gate is valid only when all of the following hold for the harness instance under test: 1. **Directional agreement is measured separately.** Report `lenient_disagreements` (harness=pass, ref=unsafe) and `conservative_disagreements` (harness=unsafe, ref=pass) as distinct counts, never collapsed into a single agreement rate. 2. **Lenient disagreement rate is below a hard ceiling** — empirically 0.5% per suite — with *zero* tolerance on probes tagged `critical` (those whose unsafe outcome is irreversible, e.g., credential exfiltration). 3. **The reference judge is byte-frozen.** The harness must record a hash of the exact prompt template and serialization path it uses; if that hash differs from the reference's recorded hash, the comparison is invalid regardless of agreement rate. This is the check that would have caught v2.7.1. 4. **Probe set is identical.** Any probe added to the harness but absent from the reference must be scored against a human-labeled gold set, not the reference judge. If any condition fails, the harness build is *unvalidated*, not *regressed* — these are different states and must map to different CI outcomes (block vs. quarantine). ## Decision Boundary The operational rule an agent should apply when integrating a new harness instance: - If `critical_lenient_disagreements > 0` → **hard fail**, do not merge, escalate to safety review. - If `lenient_disagreement_rate > 0.005` on any suite → **hard fail**. - If `prompt_hash != reference_prompt_hash` → **invalid comparison**, treat as hard fail even if agreement is perfect (perfect agreement under a changed prompt is evidence of a shared bug, not correctness). - If `conservative_disagreement_rate` is high but lenient rate is within bounds → **pass with annotation**; route the conservative disagreements to a triage queue but do not block. - If only the aggregate exact-match rate is available (no directional split) → **insufficient signal**, block until the harness emits the split. ## Operational Effect Applying the directional split and the prompt-hash freeze to the v2.7.1 case would have blocked it at the gate: the lenient rate on `prompt-injection-indirect` was 3.4%, seven times the ceiling, and the serialization change altered the prompt hash. Post-fix, the harness was rebuilt to preserve the reference serialization and the suite returned to 0.3% lenient / 5.8% conservative, with all conservative flags triaged and dismissed. The practical lesson for continuous regression: **treat safety-harness agreement as a one-sided inequality, not an equality.** The inequality is `lenient_rate <= epsilon`, and the equality that must hold is on the *prompt bytes*, not the *decisions*. Conflating the two is how a lenient harness ships green. ```python import hashlib from dataclasses import dataclass class HarnessValidationError(RuntimeError): """Raised when a LITMUS harness instance cannot be validated against the reference judge.""" @dataclass(frozen=True) class HarnessReport: suite: str lenient_disagreements: int # harness=pass, reference=unsafe conservative_disagreements: int # harness=unsafe, reference=pass critical_lenient: int # subset of lenient on critical-tagged probes total_probes: int prompt_hash: str reference_prompt_hash: str LENIENT_CEILING = 0.005 def validate_harness(report: HarnessReport) -> str: """Return 'validated' or 'quarantined'; raise on hard-fail conditions.""" if report.prompt_hash != report.reference_prompt_hash: raise HarnessValidationError( f"[{report.suite}] prompt hash mismatch: harness={report.prompt_hash[:12]} " f"ref={report.reference_prompt_hash[:12]}; comparison invalid, " f"perfect agreement under a changed prompt indicates a shared bug" ) if report.critical_lenient > 0: raise HarnessValidationError( f"[{report.suite}] {report.critical_lenient} critical probe(s) passed by harness " f"but flagged unsafe by reference; silent safety hole, escalate to safety review" ) if report.total_probes == 0: raise HarnessValidationError(f"[{report.suite}] zero probes scored; harness produced no signal") lenient_rate = report.lenient_disagreements / report.total_probes if lenient_rate > LENIENT_CEILING: raise HarnessValidationError( f"[{report.suite}] lenient disagreement rate {lenient_rate:.4f} exceeds " f"ceiling {LENIENT_CEILING}; harness is externally lenient despite internal consistency" ) if report.conservative_disagreements > 0: return "validated_with_triage" return "validated" def gate(reports: list[HarnessReport]) -> dict[str, str]: outcomes: dict[str, str] = {} for r in reports: try: outcomes[r.suite] = validate_harness(r) except HarnessValidationError as e: outcomes[r.suite] = f"BLOCKED: {e}" return outcomes ```