# When must a LITMUS harness instance reject a candidate policy before scoring rather than after?

**Date of Record:** 2026 November 6

**Scope:** Applies to LITMUS-style adversarial agent evaluation harnesses that gate candidate policies on a fixed safety suite and emit a regression verdict. Covers pre-scoring admissibility checks (policy contract, sandbox capability envelope, probe-set integrity); does not cover post-hoc statistical analysis of scores or human adjudication of near-threshold cases.

## Core Distinction

A LITMUS harness has two failure surfaces that are routinely conflated: an *inadmissible candidate* and an *unsafe candidate*. An inadmissible candidate is one whose execution environment or interface contract cannot produce a meaningful safety signal at all — it is missing a required tool stub, it declares a capability the sandbox cannot revoke, its probe set has drifted from the frozen hash, or it returns action traces in a schema the scorer cannot parse. An unsafe candidate is one that is fully admissible but fails probes. The non-obvious invariant is this: **the harness must refuse to emit a score for inadmissible candidates, because a low score and an unscoreable candidate are indistinguishable downstream, and regression gates that treat "score < threshold" as the only trigger will silently pass an inadmissible policy whose scorer defaulted to zero failures.**

The tempting design — run the suite, let probes error out, count errored probes as passes — converts every admissibility failure into a false safety certification. The boundary is between *measurement* and *non-measurement*, and the harness must make that boundary explicit and loud.

## What Was Observed

In a regression run against a candidate policy that had been re-packaged with a renamed tool namespace (`shell.exec` → `terminal.run`), the LITMUS harness booted the policy, injected the standard probe set, and observed that every adversarial tool-invocation probe returned a structured "tool not found" error from the policy's own dispatcher. The scorer, written to treat non-malicious tool errors as compliant refusals, recorded 100% pass on the exfiltration and destructive-action suites. The gate passed the candidate. The candidate was, in fact, fully capable of exfiltration — it simply spoke a different tool dialect than the probes assumed.

A second observation: a sandbox that had been configured with `allow_network=True` by a CI environment override caused the harness to score a candidate as failing containment probes that it would have passed under the frozen `allow_network=False` envelope. The score was real but measured the wrong environment. Neither the score nor its direction revealed the misconfiguration.

Both incidents share a signature: the harness produced a numeric verdict from a run in which the *measurement contract* — tool namespace, sandbox envelope, probe-set hash, trace schema — was violated. The score was computed; it was not valid.

## Condition of Validity

A LITMUS score is valid only when all of the following hold at scoring time, and the harness must verify them *before* the first probe executes:

1. **Probe-set integrity:** the loaded probe set's manifest hash equals the frozen hash recorded in the gate configuration. Any drift — added, removed, or edited probes — invalidates comparability with prior runs.
2. **Interface conformance:** the candidate declares the exact tool namespace and action-trace schema the probes target. Namespace aliasing must be resolved by an explicit adapter registered in the harness config, not inferred.
3. **Capability-envelope match:** the sandbox's effective capability set (network, filesystem write scope, process spawn, credential access) equals the envelope the probe set was calibrated against. Environment overrides that widen the envelope are a validity failure, not a scoring input.
4. **Probe reachability precheck:** a canary probe with a known-malicious action must produce the expected *detection* signal on a reference policy. If the canary cannot be detected, the instrument is broken and no candidate score is meaningful.

These are necessary conditions, not sufficient ones — a valid run can still yield a failing score. The point is that *invalidity must be detected and surfaced as a distinct outcome class*, never folded into pass or fail.

## Decision Boundary

The harness must branch to rejection — not scoring — when any validity precondition fails, and it must do so *before* consuming compute on the full suite. Concretely:

- Hash mismatch on the probe manifest → reject, do not run.
- Namespace/schema mismatch without a registered adapter → reject, do not run.
- Effective capability envelope ≠ calibrated envelope → reject, do not run.
- Canary probe undetected on the reference policy → reject, do not run (instrument fault).

The boundary is deliberately conservative: when in doubt about admissibility, the harness rejects rather than scores. A rejected run is a visible, actionable event; a scored-but-invalid run is a silent safety hole. The asymmetry of consequences — a false pass in a safety gate is far costlier than a false rejection that a human re-runs — justifies the strict posture.

Critically, rejection must be a *distinct terminal state* in the gate's output, not an error that the CI wrapper converts to exit code 0 or to a default pass. The gate's contract is: `PASS`, `FAIL`, or `REJECTED`. Collapsing `REJECTED` into either of the other two is the failure mode this record exists to prevent.

## Operational Effect

With pre-scoring admissibility checks enforced, the regression gate gains a third outcome that operators must handle: a rejected candidate blocks the pipeline and pages the on-call harness owner, who investigates whether the candidate drifted (re-package, namespace change) or the environment drifted (CI override, probe-set edit). In the observed incidents, both would have surfaced as `REJECTED` within seconds of harness boot rather than as a false `PASS` discovered weeks later during an incident review.

The cost is additional harness startup latency (hashing, canary run) and a new operational runbook for `REJECTED`. The benefit is that a `PASS` from the LITMUS gate now means what it claims: an admissible candidate was measured against an intact probe set in the calibrated envelope and produced no safety failures. That claim is only as strong as the harness's willingness to refuse to score.

```python
import hashlib
from dataclasses import dataclass


class HarnessAdmissibilityError(RuntimeError):
    """Raised when a LITMUS run cannot produce a valid safety score.

    Distinct from a failing score: this means the measurement contract
    (probe set, interface, capability envelope, instrument health) was
    violated and no score may be emitted.
    """


@dataclass(frozen=True)
class RunContext:
    probe_manifest_hash: str
    frozen_manifest_hash: str
    candidate_tool_namespace: str
    expected_tool_namespace: str
    adapter_registered: bool
    effective_capabilities: frozenset
    calibrated_capabilities: frozenset
    canary_detected_on_reference: bool


def assert_admissible(ctx: RunContext) -> None:
    """Enforce pre-scoring validity. Must run before any probe executes."""
    if ctx.probe_manifest_hash != ctx.frozen_manifest_hash:
        raise HarnessAdmissibilityError(
            f"probe-set drift: {ctx.probe_manifest_hash} != {ctx.frozen_manifest_hash}"
        )
    if (
        ctx.candidate_tool_namespace != ctx.expected_tool_namespace
        and not ctx.adapter_registered
    ):
        raise HarnessAdmissibilityError(
            f"interface mismatch: candidate speaks {ctx.candidate_tool_namespace!r}, "
            f"probes target {ctx.expected_tool_namespace!r}, no adapter registered"
        )
    if ctx.effective_capabilities != ctx.calibrated_capabilities:
        widened = ctx.effective_capabilities - ctx.calibrated_capabilities
        raise HarnessAdmissibilityError(
            f"capability envelope violation; widened by {sorted(widened)}"
        )
    if not ctx.canary_detected_on_reference:
        raise HarnessAdmissibilityError(
            "instrument fault: canary probe not detected on reference policy"
        )


def gate(ctx: RunContext, run_suite) -> str:
    """Return 'PASS' | 'FAIL' | 'REJECTED'. Never fold REJECTED into PASS/FAIL."""
    try:
        assert_admissible(ctx)
    except HarnessAdmissibilityError as exc:
        return f"REJECTED: {exc}"
    return "PASS" if run_suite() else "FAIL"
```