# When must an instance refuse to start: Harness/Agent Version Skew in LITMUS CI Red-Teaming Pipelines

**Date of Record:** 2026 November 10

**Scope:** Applies to LITMUS safety-benchmark harness integrations where a CI job instantiates a target agent (or a versioned agent image) and drives it through a fixed corpus of adversarial probes. Does not apply to offline batch scoring of pre-recorded transcripts, nor to harnesses that pin both agent and probe-set to a single immutable artifact digest.

## Core Distinction

The non-obvious invariant is this: **a LITMUS probe corpus is only meaningful relative to a specific *harness contract version*, not merely relative to an agent version.** A CI pipeline that gates merges on "zero safety regressions" is comparing two numbers that were produced under two different contracts. The failure is not that the agent got less safe; it is that the *probe semantics* shifted underneath the comparison while the harness binary was upgraded independently of the probe-set manifest.

The trap is that both artifacts usually carry semver, and semver *looks* compatible. A harness bump from `2.7.1` to `2.8.0` (minor, non-breaking by semver) can change how a probe's `expected_refusal` field is interpreted — for example, from "must produce a refusal token" to "must not produce a compliant-token sequence." Same probe ID, same corpus hash, different verdict function. The agent under test is unchanged; the *scoring* changed; the gate flips.

## What Was Observed

In a representative pipeline, the LITMUS harness is invoked per-PR as:

```
litmus run --agent-image $IMAGE --corpus $CORPUS_HASH --harness $HARNESS_TAG --out results.json
```

The gate compares `results.json` against a baseline produced on `main`. During a routine harness upgrade, the baseline was *not* regenerated. The observed sequence:

1. Harness `2.7.1` → `2.8.0` landed on `main` with a changelog entry reading "clarify refusal scoring for multi-turn probes."
2. Per-PR runs began using `2.8.0` against the *old* baseline from `2.7.1`.
3. Three agent PRs that touched only tool-call formatting were blocked as "safety regressions."
4. Root cause: multi-turn probes that previously scored `PASS` on a terminal refusal now scored `FAIL` because `2.8.0` required the refusal to occur on *every* turn, not just the final one. The agents were behaviorally identical on those probes; the verdict function was stricter.
5. A fourth PR — which genuinely regressed on a single-turn jailbreak — was *masked* because the stricter multi-turn scoring inflated the baseline's apparent failure count, pushing the delta below the gate's absolute threshold.

This is the dangerous case: version skew produced a **false positive and a false negative simultaneously**, in the same run, from the same contract change.

## Condition of Validity

The gate is only valid when the following holds for every comparison:

- `corpus_hash` is identical between run and baseline (necessary, not sufficient — this was satisfied and the failure still occurred).
- `harness_contract_version` is identical between run and baseline. This is **not** the harness package version; it is the explicit contract identifier the harness emits into `results.json` (e.g., `"contract": "refusal-scoring/v3"`). Minor package bumps may or may not change it; the pipeline must read it, not infer it.
- The verdict function is deterministic given `(corpus_hash, contract_version, agent_response)`. If the harness introduces any nondeterminism (e.g., LLM-as-judge for refusal classification), the baseline must be regenerated on the same judge model *and* judge prompt hash.

When any of these do not hold, the comparison is *undefined*, not *failed*. The correct pipeline behavior is to refuse to produce a verdict — i.e., the run must not start (or must abort before emitting a gate result), rather than emit `PASS`/`FAIL` against a mismatched baseline.

## Decision Boundary

The boundary is precise: **an instance must refuse to start the comparison — not the agent, the *gate* — when `contract_version` in the fresh run differs from `contract_version` recorded in the baseline manifest.** Treating this as a soft warning is the failure mode. The gate must hard-fail closed with a distinct, non-retryable error class, because retrying will never resolve it: the fix is a human decision (regenerate baseline, or pin harness to the baseline's contract).

Secondary boundary: when the harness emits a contract version *newer* than the pipeline's baseline schema knows how to parse, the pipeline must also refuse — silently dropping unknown fields (e.g., a new `turn_level_refusals` array) reintroduces the masking failure above, because the delta computation ignores the new signal.

Do *not* apply this refusal when the agent image is the only thing that changed and `contract_version` and `corpus_hash` both match; that is the legitimate regression-detection case and must proceed.

## Operational Effect

After the incident, the pipeline was changed to:

1. Emit and persist `contract_version` in every `results.json` and every baseline manifest.
2. Compare `(corpus_hash, contract_version)` as a tuple before any delta computation; mismatch → `HarnessContractMismatch`, exit non-zero, no gate verdict.
3. Require baseline regeneration as an explicit, reviewed step whenever the harness changelog touches scoring, and fail CI if a harness bump lands without a corresponding baseline manifest update (detected by comparing the harness's declared contract version against the baseline's).
4. Distinguish exit codes: `2` = safety regression (block merge, agent issue), `3` = contract mismatch (block merge, *infrastructure* issue, route to harness owners). Conflating these sent three agent PRs to the wrong team.

The general lesson: in continuous red-teaming, the *scoring contract* is a first-class versioned artifact, and any pipeline that compares verdicts across contract versions is measuring noise. Fail closed on skew.

```
```python
class HarnessContractMismatch(RuntimeError):
    """Raised when a LITMUS run's scoring contract does not match the baseline's.

    This is an infrastructure fault, not a safety regression. It must not be
    reported as PASS or FAIL; the comparison is undefined.
    """
    pass


def assert_comparable(fresh_result: dict, baseline_manifest: dict) -> None:
    fresh_contract = fresh_result.get("contract_version")
    base_contract = baseline_manifest.get("contract_version")
    fresh_corpus = fresh_result.get("corpus_hash")
    base_corpus = baseline_manifest.get("corpus_hash")

    if fresh_corpus != base_corpus:
        raise HarnessContractMismatch(
            f"corpus hash mismatch: run={fresh_corpus!r} baseline={base_corpus!r}; "
            "regenerate baseline against current corpus"
        )

    if fresh_contract != base_contract:
        raise HarnessContractMismatch(
            f"scoring contract mismatch: run={fresh_contract!r} "
            f"baseline={base_contract!r}; verdict deltas are undefined. "
            "Regenerate the baseline under the new contract, or pin the harness "
            "to the baseline's contract version. Do not emit a gate verdict."
        )

    if fresh_contract is None:
        raise HarnessContractMismatch(
            "harness did not emit contract_version; refusing to compare "
            "against an unversioned baseline"
        )


def compute_gate_verdict(fresh_result: dict, baseline_manifest: dict) -> str:
    assert_comparable(fresh_result, baseline_manifest)
    delta = fresh_result["failure_count"] - baseline_manifest["failure_count"]
    return "REGRESSION" if delta > 0 else "OK"
```