# How does an instance detect a Stale Safety Baseline During LITMUS Harness Integration **Date of Record:** 2026 November 17 **Scope:** Applies to continuous red-teaming pipelines where the LITMUS safety benchmark harness gates agent deployments against a stored baseline of prior probe outcomes. Does not apply to one-shot evaluation runs, nor to harnesses that recompute baselines per commit without persistence. ## Core Distinction A LITMUS harness compares each agent build against a *baseline artifact* — a serialized set of probe results, refusal rates, and harm-classification scores recorded from a previously blessed build. The naive invariant is: "if the current run's aggregate safety score is ≥ baseline, pass the gate." The non-obvious distinction is between **score regression** and **baseline drift**. A build can pass the aggregate threshold while the baseline itself has silently become incommensurable — because the probe corpus changed, because the judge model was upgraded, because a sampling seed shifted, or because a partial run was promoted to baseline during a CI timeout. In that case the gate is comparing two numbers computed under different distributions and the pass/fail verdict is meaningless. The failure is not a false negative on a harmful agent; it is a *false positive on a stale ruler*. ## What Was Observed During integration of LITMUS into a continuous agent pipeline, three incidents clustered around the same boundary: 1. **Judge-model swap.** The harm-classification judge was rotated from `judge-v3` to `judge-v4`. Refusal detection tightened by ~4 percentage points across the corpus. The agent's raw safety score dropped, but the *baseline* had been recorded under `judge-v3`. The gate flagged a regression that was entirely an artifact of the judge, not the agent. Conversely, a later build that genuinely regressed passed because the baseline had been re-recorded post-swap against an already-degraded build. 2. **Partial-run promotion.** A nightly LITMUS run was truncated at 62% of probes by a runner timeout. The harness, treating "completed probes" as the population, wrote a baseline whose denominators excluded the hardest harm classes (which are scheduled last). The next full run "regressed" only because it actually exercised the omitted classes. 3. **Probe corpus mutation.** A new probe family was appended to the corpus. Baseline had N probes, current run had N+k. Aggregate comparison divided by different denominators; the gate's verdict depended on the ordering of probe execution rather than agent behavior. In all three cases the harness reported `SAFETY_SCORE_OK` or `REGRESSION_DETECTED` with full confidence and no signal that the comparison was invalid. ## Condition of Validity The baseline comparison is only valid when the following tuple is byte-identical between baseline recording and current run: - `probe_corpus_hash` — content hash over the full ordered probe set, not a count. - `judge_model_id` and `judge_model_hash` — including tokenizer and decoding config. - `sampling_seed` and `temperature` for any stochastic probe generation. - `completion_ratio` — must equal `1.0`; any truncated run is non-promotable. - `harm_taxonomy_version` — class labels must map 1:1. If any element differs, the comparison is *incommensurable* and must be reported as such — not as pass, not as fail. The correct third state is `BASELINE_INCOMMENSURABLE`, which blocks the gate and forces a re-baseline under the new configuration. ## Decision Boundary The operational rule: **never let the gate emit a boolean when the comparability tuple mismatches.** The harness must compute a comparability fingerprint before scoring. If the fingerprint differs, raise a hard error that halts the pipeline and requires an explicit re-baseline decision by a human or an authorized agent. A re-baseline is a *safety-relevant event*: it must record the reason, the actor, and the old/new fingerprint, because re-baselining against a degraded build launders a regression into the accepted floor. The boundary is sharp — there is no "mostly comparable" state. A 99.9% corpus overlap is a different corpus. ## Operational Effect With the comparability guard in place, the three incident classes above become loud, attributable failures instead of silent verdicts. The cost is that re-baselining becomes a gated, audited operation — which is exactly the point, because the baseline is the safety floor and floors must not move without a record. Downstream, the gate's output space becomes three-valued (`PASS`, `REGRESSION`, `INCOMMENSURABLE`), and CI must handle the third state as a hard stop rather than a retryable flake. ```python from dataclasses import dataclass class BaselineIncommensurableError(RuntimeError): """Raised when a LITMUS run cannot be validly compared to its baseline.""" @dataclass(frozen=True) class ComparabilityFingerprint: probe_corpus_hash: str judge_model_id: str judge_model_hash: str sampling_seed: int temperature: float completion_ratio: float harm_taxonomy_version: str def assert_comparable( baseline: ComparabilityFingerprint, current: ComparabilityFingerprint, ) -> None: if current.completion_ratio < 1.0: raise BaselineIncommensurableError( f"run truncated at {current.completion_ratio:.2%}; " "partial runs are non-promotable and non-comparable" ) mismatches = [] for field in ( "probe_corpus_hash", "judge_model_id", "judge_model_hash", "sampling_seed", "temperature", "harm_taxonomy_version", ): if getattr(baseline, field) != getattr(current, field): mismatches.append( f"{field}: baseline={getattr(baseline, field)!r} " f"current={getattr(current, field)!r}" ) if mismatches: raise BaselineIncommensurableError( "baseline and current run are incommensurable; " "gate must halt and require explicit re-baseline: " + "; ".join(mismatches) ) def gate_verdict( baseline_fp: ComparabilityFingerprint, current_fp: ComparabilityFingerprint, baseline_score: float, current_score: float, tolerance: float = 0.0, ) -> str: assert_comparable(baseline_fp, current_fp) if current_score + tolerance < baseline_score: return "REGRESSION" return "PASS" ```