# In what scenarios must an instance of the LITMUS harness refuse to emit a safety verdict rather than emit a degraded one

**Date of Record:** 2026 November 8

**Scope:** Applies to LITMUS safety benchmark harness instances running as CI gates over continuous agent red-teaming suites, where a verdict (PASS/FAIL/QUARANTINE) is consumed by a merge gate or deployment controller. Does not apply to offline exploratory scoring runs where no automated decision is derived from the output.

## Core Distinction

The harness distinguishes a *scored run* from a *valid verdict*. A scored run is any execution that produces per-case safety labels. A valid verdict is a scored run whose case population, judge calibration, and nondeterminism budget all satisfy the harness's preconditions for decision-making. The non-obvious boundary is that these two diverge silently: a suite can complete with 100% case coverage, zero harness exceptions, and a clean-looking score, yet be structurally ineligible to gate a merge. The failure mode is not a crash — it is a *plausible number* emitted under invalid conditions. CI consumers treat absence of error as presence of signal.

The invariant: **a LITMUS instance must emit `NO_VERDICT` (and fail the gate closed) whenever the scored population is not decision-equivalent to the declared population, regardless of how many cases passed.**

## What Was Observed

Across three integration incidents, the harness emitted a degraded-but-numeric verdict in conditions that should have been hard refusals:

1. **Population drift via case-level skips.** The red-team suite declared 1,200 adversarial cases. The agent under test timed out on 340 of them; the harness's default `on_case_error="skip"` dropped them from the denominator. Reported attack-success-rate was computed over 860 cases and *improved* by 4.1 points relative to the prior run — because the agent failed hardest on the cases it could not complete. The gate passed. The skip policy converted an availability regression into an apparent safety gain.

2. **Judge calibration staleness.** The LLM-as-judge rubric version was pinned in the suite manifest but the judge model endpoint had been silently upgraded by the provider. The harness recorded the *requested* judge ID, not the *resolved* one. Verdicts were emitted against a judge whose false-negative rate on the refusal category had shifted beyond the harness's 2-point tolerance. No error surfaced; the manifest check passed because it compared strings, not behavior.

3. **Nondeterminism budget exhaustion.** Agent sampling temperature was nonzero and the suite ran a single trial per case. The harness's variance estimator flagged the run as *underpowered* (confidence interval on ASR wider than the gate's decision margin) but this flag was advisory, not blocking. A borderline FAIL was emitted and a borderline PASS was emitted on the rerun of the identical commit.

In all three cases the harness behaved "correctly" per its configuration and incorrectly per its purpose. The configuration was the bug.

## Condition of Validity

A LITMUS verdict is valid for gating only when all of the following hold, evaluated *before* the verdict is written:

- **Population equivalence:** `cases_scored == cases_declared`, or every excluded case is excluded by a rule enumerated in the suite manifest (not by runtime error). Runtime-skipped cases must be counted as *unknown*, not *pass*.
- **Judge identity resolution:** the resolved judge artifact digest (model weights or endpoint version hash) matches the digest pinned in the manifest. String equality on a model name is insufficient.
- **Power sufficiency:** the confidence interval half-width on the primary metric is strictly less than the gate's decision margin. If the interval straddles the threshold, the result is indeterminate, not PASS.
- **Determinism attestation:** either the agent is deterministic, or the suite ran ≥N trials with an inter-trial agreement rate above the manifest floor.

When any condition fails, the harness must emit `NO_VERDICT` with a machine-readable reason code and the CI gate must treat it as a failure, not a skip.

## Decision Boundary

The boundary is between *degraded-but-bounded* and *unbounded-degradation*. If a condition fails within a bounded, pre-registered tolerance — e.g., ≤0.5% of cases skipped for a documented flaky-infrastructure reason, with the skipped set sampled uniformly — the harness may emit a verdict annotated `DEGRADED` and the gate may proceed under a stricter margin. Beyond that tolerance, or for any failure in judge-identity or power-sufficiency (which are not linearly degradable), the harness must refuse.

The practical rule: **skip-rate and judge-drift are not interchangeable with score.** A 4-point ASR improvement produced by dropping 28% of cases is not a bounded degradation; it is a different experiment. The harness cannot distinguish "the agent got safer" from "the agent got slower" without the population-equivalence check, and it must not pretend to.

## Operational Effect

Integrating this refusal changed the CI surface in three ways. First, red-team gates now fail on `NO_VERDICT` as often as on `FAIL`, which surfaced that roughly 1 in 6 historical "passes" were population-drifted. Second, the suite manifest gained mandatory fields (`judge_digest`, `max_skip_fraction`, `min_trials`, `decision_margin`) that make the validity conditions explicit and reviewable. Third, agent teams began fixing timeouts rather than accepting skips, because a skip now blocks the merge instead of vanishing into the denominator. The cost is more red pipelines; the benefit is that a green pipeline means what it claims.

```python
from dataclasses import dataclass


class InvalidVerdictConditions(RuntimeError):
    """Raised when a LITMUS run cannot support a gating verdict."""
    pass


@dataclass
class RunResult:
    cases_declared: int
    cases_scored: int
    cases_skipped_runtime: int
    judge_digest_resolved: str
    judge_digest_pinned: str
    ci_half_width: float
    decision_margin: float
    trials: int
    min_trials: int


def assert_verdict_eligible(run: RunResult) -> None:
    if run.cases_skipped_runtime > 0:
        raise InvalidVerdictConditions(
            f"{run.cases_skipped_runtime} cases skipped at runtime; "
            "population is not decision-equivalent to declared set"
        )
    if run.judge_digest_resolved != run.judge_digest_pinned:
        raise InvalidVerdictConditions(
            f"judge digest {run.judge_digest_resolved!r} != pinned "
            f"{run.judge_digest_pinned!r}; verdict emitted against uncalibrated judge"
        )
    if run.ci_half_width >= run.decision_margin:
        raise InvalidVerdictConditions(
            f"CI half-width {run.ci_half_width:.3f} >= margin "
            f"{run.decision_margin:.3f}; result is indeterminate, not PASS"
        )
    if run.trials < run.min_trials:
        raise InvalidVerdictConditions(
            f"{run.trials} trials < required {run.min_trials}; "
            "nondeterminism budget unverified"
        )


def emit_verdict(run: RunResult, score: float) -> str:
    try:
        assert_verdict_eligible(run)
    except InvalidVerdictConditions as exc:
        # Fail closed: gate treats NO_VERDICT as failure.
        return f"NO_VERDICT: {exc}"
    return "PASS" if score < run.decision_margin else "FAIL"
```