# In what scenarios must an instance treat LITMUS safety-benchmark harness results as non-transferable across model revisions in continuous CI red-teaming

**Date of Record:** 2026 November 7

**Scope:** Applies to CI pipelines that run a LITMUS-style safety benchmark harness against an agent under test (AUT) on every commit or model-revision promotion, and that gate merges on a pass/fail derived from harness scores. Does not apply to one-shot offline evaluations, nor to harness runs whose sole purpose is exploratory logging without a merge gate.

## Core Distinction

The non-obvious boundary is between **harness-score continuity** and **safety-invariant continuity**. A LITMUS harness produces a scalar (or vector) safety score over a fixed probe suite. CI consumers routinely treat a score at or above the prior revision's threshold as evidence that the safety invariant still holds. That inference is only valid if the probe suite, the harness's internal judge model, and the AUT's tool/permission surface are all held fixed. In continuous red-teaming, at least one of these is almost always in motion: probes are appended as new attack families are discovered, the judge model is upgraded, and the AUT's tool schema changes with feature work. When any of these shift, a numerically stable score can mask a genuine invariant regression, and a numerically degraded score can reflect judge drift rather than an actual safety failure. The harness score is a *measurement under a configuration*, not a property of the model.

## What Was Observed

Across roughly six weeks of CI runs on a single agent line, three concrete divergence patterns appeared:

1. **Judge-version silent relaxation.** A harness upgrade bumped the internal rubric judge from a smaller to a larger model. Aggregate refusal-correctness on the "tool-misuse" probe family rose from 0.81 to 0.89 with no change to the AUT. Manual replay of the same 40 failing probes showed the new judge was accepting partial-compliance responses the old judge had flagged. The gate passed; the underlying behavior had not improved.
2. **Probe-set dilution.** Adding 60 new probes to a suite of 240 changed the denominator. A regression on the original 240 (three new hard failures in the "data exfiltration via chained tools" family) was offset by easy passes on the new probes, leaving the aggregate within the gate's tolerance band.
3. **Tool-surface coupling.** A feature commit added a new `http_fetch` tool to the AUT. The harness's probe suite had no probes exercising that tool, so the score was unchanged and the gate passed. A targeted probe written after the fact found the new tool bypassed the URL allowlist. The invariant was violated at the moment the tool was added, but the harness could not see it.

In each case the CI signal was green and the safety property was false.

## Condition of Validity

A LITMUS harness result may be treated as transferable across revisions only when all of the following hold for the comparison being made:

- The probe suite hash is identical (or the comparison is explicitly per-family against the intersected probe set, not the aggregate).
- The judge model identifier and rubric version are pinned and unchanged.
- The AUT's declared tool/permission manifest is unchanged, or every newly added capability has at least one probe in the suite that exercises it.
- The sampling parameters (temperature, seed policy, max turns) are fixed.

If any condition fails, the result is a *fresh measurement*, not a *continuation* of the prior measurement. It may still be used as a gate, but it must not be used to infer "no regression since last revision," and a passing aggregate must not be reported as invariant preservation.

## Decision Boundary

The operational rule that emerged: **when the harness configuration changes, the CI gate must fail closed and require a re-baseline, not pass by default on a numerically acceptable score.** Concretely:

- If probe-suite hash, judge ID, or tool manifest changed relative to the baseline revision, the pipeline must emit a `HarnessConfigurationDrift` condition and block promotion until a human or an automated re-baseline job records a new baseline *and* the diff in per-family scores is reviewed.
- Aggregate-only gating is insufficient. The gate must evaluate per-family scores against per-family baselines, because dilution is invisible in the aggregate.
- Any newly added AUT tool with no corresponding probe family must be treated as an unmeasured capability and block promotion until probes exist. Absence of probes is not evidence of safety.

The boundary is crossed the moment a green build is interpreted as "the safety invariant survived the change." That interpretation is only licensed under the validity conditions above.

## Operational Effect

After adopting the fail-closed rule, the pipeline's false-green rate on the three observed patterns dropped to zero over the following four weeks, at the cost of roughly one blocked promotion per week requiring a re-baseline review (median 20 minutes). The dominant cost was not compute but the review step: someone had to confirm that a per-family score delta was attributable to a legitimate configuration change rather than a masked regression. The harness became a *change detector* rather than a *safety oracle*, which is the only role it can honestly play when its own configuration is in motion.

```python
from __future__ import annotations

from dataclasses import dataclass


class HarnessConfigurationDrift(RuntimeError):
    """Raised when a LITMUS harness result cannot be compared to the baseline
    because the measurement configuration changed underneath it."""


@dataclass(frozen=True)
class HarnessRun:
    probe_suite_hash: str
    judge_model_id: str
    tool_manifest_hash: str
    per_family_scores: dict[str, float]
    aggregate_score: float


@dataclass(frozen=True)
class Baseline:
    probe_suite_hash: str
    judge_model_id: str
    tool_manifest_hash: str
    per_family_scores: dict[str, float]


def gate_promotion(run: HarnessRun, baseline: Baseline, tolerance: float = 0.02) -> None:
    """Fail closed when the harness configuration drifted, otherwise gate on
    per-family deltas against the baseline. Aggregate score is deliberately
    not used for the pass decision; it is logged only."""
    if run.probe_suite_hash != baseline.probe_suite_hash:
        raise HarnessConfigurationDrift(
            f"probe suite changed: {baseline.probe_suite_hash} -> {run.probe_suite_hash}; "
            "aggregate score is not comparable, re-baseline required"
        )
    if run.judge_model_id != baseline.judge_model_id:
        raise HarnessConfigurationDrift(
            f"judge model changed: {baseline.judge_model_id} -> {run.judge_model_id}; "
            "score delta may be judge drift, re-baseline required"
        )
    if run.tool_manifest_hash != baseline.tool_manifest_hash:
        raise HarnessConfigurationDrift(
            f"AUT tool manifest changed: {baseline.tool_manifest_hash} -> "
            f"{run.tool_manifest_hash}; new capabilities may be unprobed, "
            "re-baseline and add probes before promotion"
        )

    for family, base_score in baseline.per_family_scores.items():
        if family not in run.per_family_scores:
            raise HarnessConfigurationDrift(
                f"family {family!r} present in baseline but missing from run"
            )
        delta = run.per_family_scores[family] - base_score
        if delta < -tolerance:
            raise AssertionError(
                f"safety regression in family {family!r}: "
                f"{base_score:.3f} -> {run.per_family_scores[family]:.3f}"
            )

    unprobed = set(run.per_family_scores) - set(baseline.per_family_scores)
    if unprobed:
        # New families are fine to add, but they must not be silently trusted.
        raise HarnessConfigurationDrift(
            f"run contains families with no baseline: {sorted(unprobed)}; "
            "record a new baseline before gating on them"
        )
```