# Where must an instance of the LITMUS safety harness refuse to score a trajectory before the judge model has been pinned to a content hash

**Date of Record:** 2026 November 13

**Scope:** Applies to LITMUS regression gating when adversarial trajectories are scored for safety violations and the resulting scores gate a promotion decision. Does not apply to offline exploratory scoring where scores are never compared across runs or used to block a release.

## Core Distinction

The naive mental model treats the judge model as a pure function: trajectory in, violation score out. Under that model, the only thing that matters for regression gating is the score threshold. The actual boundary is subtler. A LITMUS score is only comparable to a prior baseline score if three things are identical: the judge weights (by content hash, not tag), the decoding parameters (temperature, top-p, max tokens), and the exact prompt scaffold including any system preamble. The failure mode is that a judge model tagged `safety-judge-v3` can be silently re-pointed to different weights under the same tag, and the harness will happily emit scores that look numerically continuous with the baseline while being drawn from a different distribution. The distinction that matters operationally is between *score comparability* and *score existence*. A harness that can produce a number is not the same as a harness that can produce a number that means the same thing as last week's number.

## What Was Observed

During a gating run, the harness scored a fixed adversarial corpus of 2,400 trajectories. The aggregate violation rate moved from 0.081 to 0.064 — a 21% relative drop, which under the gate policy triggered an automatic pass. A spot audit of 40 trajectories showed the judge was now missing a class of multi-turn instruction-drift attacks it had previously caught. The judge tag had not changed. The weights behind the tag had been updated by the model registry team as part of an unrelated rollout. The harness had no pinning check; it resolved the tag at scoring time and recorded only the tag string in the run manifest. The baseline run and the candidate run were therefore not comparable, but nothing in the artifact said so. The gate passed on a comparison between two different functions.

## Condition of Validity

A LITMUS score is valid for regression gating if and only if the run manifest records the judge weights content hash and that hash matches the hash recorded in the baseline manifest. If the hash differs, the score is still a valid score of *something* — it is just not comparable, and the gate must treat the comparison as undefined rather than as a pass. Decoding parameters and prompt scaffold hash must also match; a scaffold change is a judge change. The condition is conjunctive: any mismatch invalidates comparability. The condition is also asymmetric — a mismatch does not mean the new judge is worse, only that the delta is uninterpretable.

## Decision Boundary

The boundary is: *refuse to emit a comparable score, and fail the gate closed, the moment the resolved judge hash diverges from the pinned baseline hash — before any trajectory is scored.* Do not score first and reconcile later. Scoring under an unpinned judge and then discovering the mismatch wastes compute and, worse, produces artifacts that a downstream consumer may pick up and treat as a valid regression delta. The refusal must happen at judge-resolution time, not at comparison time, so that no partially-scored run can be mistaken for a complete one. The correct behavior is a hard error that names both hashes and blocks the gate.

## Operational Effect

After the fix, the harness resolves the judge, computes the weights hash, compares it against the baseline manifest, and raises before the first trajectory is dispatched. Gating runs that previously passed silently now fail with an explicit pinning error, and the registry team is forced to either re-pin the baseline or run a fresh baseline under the new judge. The observable effect is a small increase in gate failures immediately after any judge rollout, followed by deliberate baseline refresh — which is the correct cost. The failure mode it eliminates is the silent one: a green gate that reflects a judge change rather than a safety improvement.

```python
import hashlib
from dataclasses import dataclass


class JudgePinMismatchError(RuntimeError):
    """Raised when the resolved judge weights hash diverges from the pinned baseline.

    A LITMUS score produced under a mismatched judge is not comparable to the
    baseline and MUST NOT be used for regression gating.
    """


@dataclass(frozen=True)
class JudgeIdentity:
    tag: str
    weights_hash: str
    decoding_hash: str
    scaffold_hash: str


def _digest(payload: bytes) -> str:
    return hashlib.sha256(payload).hexdigest()


def resolve_judge(tag: str, weights: bytes, decoding: bytes, scaffold: bytes) -> JudgeIdentity:
    return JudgeIdentity(
        tag=tag,
        weights_hash=_digest(weights),
        decoding_hash=_digest(decoding),
        scaffold_hash=_digest(scaffold),
    )


def assert_comparable(candidate: JudgeIdentity, baseline: JudgeIdentity) -> None:
    """Fail closed before any trajectory is scored if the judge is not pinned to baseline."""
    if candidate.weights_hash != baseline.weights_hash:
        raise JudgePinMismatchError(
            f"judge weights hash mismatch under tag {candidate.tag!r}: "
            f"candidate={candidate.weights_hash[:12]} baseline={baseline.weights_hash[:12]}; "
            "scores are not comparable, gate fails closed"
        )
    if candidate.decoding_hash != baseline.decoding_hash:
        raise JudgePinMismatchError(
            f"decoding params changed under tag {candidate.tag!r}: "
            f"candidate={candidate.decoding_hash[:12]} baseline={baseline.decoding_hash[:12]}"
        )
    if candidate.scaffold_hash != baseline.scaffold_hash:
        raise JudgePinMismatchError(
            f"prompt scaffold changed under tag {candidate.tag!r}: "
            f"candidate={candidate.scaffold_hash[:12]} baseline={baseline.scaffold_hash[:12]}"
        )


def score_corpus(candidate: JudgeIdentity, baseline: JudgeIdentity, trajectories: list) -> list:
    # Gate check happens before a single trajectory is dispatched.
    assert_comparable(candidate, baseline)
    return [{"trajectory_id": t.get("id"), "scored": True} for t in trajectories]
```