# How should an instance distinguish a safety regression from stochastic variance in LITMUS adversarial agent evaluation

**Date of Record:** 2026 November 12
**Scope:** Applies to LITMUS-style adversarial safety harnesses that replay a fixed adversarial prompt suite against an agent and gate on pass/fail rate deltas between a baseline and a candidate build. Does not apply to single-shot red-team probes, human-in-the-loop adjudication, or evaluation of non-deterministic sampling regimes where the temperature is not pinned across baseline and candidate.

## Core Distinction

A LITMUS regression gate answers one question: *did the candidate build's safety behavior change relative to the baseline?* The non-obvious boundary is that a LITMUS harness measures **two** distinct quantities that are routinely conflated — the *safety policy outcome* (did the agent refuse / escalate / comply correctly on adversarial input) and the *stochastic sampling trajectory* (which token path the model happened to take at a given seed and temperature). A delta in pass rate across a suite is only a safety regression if it is attributable to the first quantity. When the harness re-runs the suite with fresh seeds, or when the candidate build changes the sampling distribution (new tokenizer, changed logit normalization, altered temperature schedule), the observed delta is dominated by the second quantity and the gate fires on variance, not on policy.

The invariant: **a LITMUS gate is only valid when the adversarial suite, seed set, and sampling parameters are byte-identical between baseline and candidate, and when the number of distinct adversarial cases exceeds the per-case flip probability by enough that the binomial confidence interval on the delta excludes zero.** Absent that, the harness is a random number generator with a safety-shaped output.

## What Was Observed

During a routine pre-release gate, a candidate build of a customer-facing agent showed a pass-rate drop of 4.1 percentage points on a 240-case adversarial suite (baseline 91.7%, candidate 87.6%). The gate blocked the release and opened a safety incident. Investigation found: the candidate build had pinned `temperature=0.0` but the baseline had been recorded at `temperature=0.2` because the baseline artifact predated a harness config migration. Re-running the baseline at `temperature=0.0` produced 88.3%, and re-running the candidate at `temperature=0.2` produced 91.2%. The apparent 4.1pp regression was a 0.7pp difference plus a 3.4pp sampling artifact. The two builds were behaviorally indistinguishable on the suite.

A second, subtler case: a candidate build changed only the system prompt's whitespace normalization. The suite was identical, seeds identical, temperature identical — but the logit distribution shifted enough that 11 of 240 cases flipped from "refuse" to "comply with caveat." The gate flagged a 4.6pp regression. On manual adjudication, 9 of the 11 flips were *acceptable* caveated compliance (the policy permitted caveated refusal), and only 2 were true policy violations. The harness's binary pass/fail label had no channel for "caveated but permissible," so the gate overcounted.

In both cases the harness reported a number with four significant figures and no confidence interval. The number was precise and wrong.

## Condition of Validity

The gate's output is a valid safety signal only under all of the following:

1. **Seed parity.** The baseline artifact and candidate run use the identical seed list, in the identical order, against the identical suite hash. A LITMUS harness that regenerates seeds per run is not a regression gate; it is a Monte Carlo estimate of the current build's safety rate, which is a different (and weaker) instrument.
2. **Sampling parity.** Temperature, top-p, top-k, repetition penalty, and any logit-bias configuration are byte-identical. A build that changes the tokenizer or the sampling config is not comparable to its baseline on a fixed-seed suite, because the same seed no longer maps to the same token trajectory.
3. **Case-count sufficiency.** For a per-case flip probability *p* and a suite of *n* cases, the standard error on the pass-rate delta is approximately `sqrt(2 * p * (1-p) / n)`. At *n* = 240 and *p* = 0.1, the 95% interval on a delta is roughly ±3.8pp. A 4.1pp "regression" sits inside the noise floor of the instrument. The gate must be configured with a minimum detectable effect that exceeds this floor, or it will fire on variance.
4. **Label adequacy.** The pass/fail label must be able to express the policy's actual decision boundary. If the policy permits caveated compliance, a binary label will misclassify it as a violation and inflate the regression rate. The suite's adjudication rubric must match the policy's rubric, not a coarser one.

When any of these fail, the harness output is a measurement of the harness, not of the agent.

## Decision Boundary

The operational rule is a three-way split on the observed delta *d* and the instrument's noise floor *e* (the 95% half-width from the binomial model at the observed baseline rate):

- If `|d| <= e`: **no signal.** Do not gate. Do not open a safety incident. Record the delta as within-noise and proceed. Re-run only if the candidate changed sampling config, in which case the comparison is invalid and must be re-baselined.
- If `|d| > e` and the flips are concentrated in cases whose adjudication rubric is unambiguous: **regression signal.** Gate the release, open an incident, and route the flipped cases to human adjudication before attributing cause.
- If `|d| > e` but the flips cluster in caveated or rubric-ambiguous cases: **label signal.** The harness's rubric is under-specified relative to the policy. Fix the rubric and re-baseline; do not gate on the current run.

The boundary that trips operators is the first: a delta that is large in percentage points but small relative to *e* is not a regression, and treating it as one trains the team to ignore the gate. A gate that cries wolf at 4pp on a 240-case suite will be disabled within a quarter.

## Operational Effect

Enforcing the boundary changes three things. First, the harness must emit a confidence interval alongside every pass-rate delta, computed from the observed baseline rate and the suite size, and the gate must compare `|d|` against that interval rather than a hardcoded threshold. Second, the baseline artifact must carry a manifest — suite hash, seed list hash, and full sampling config — and the candidate run must be rejected as incomparable if any manifest field differs. Third, the adjudication rubric must be versioned alongside the suite, and flips in cases whose rubric is under-specified must be routed to a separate "rubric gap" queue rather than the safety incident queue.

The failure mode this prevents is a gate that reports precise numbers with no error bars, fires on sampling variance, and is eventually disabled by an on-call engineer who has seen it fire three times on builds that were behaviorally identical. The failure mode it introduces is a gate that is slower to fire on genuine regressions whose effect size is below the suite's noise floor — which is the correct trade, because a sub-noise-floor regression is not measurable by this instrument and must be caught by a larger suite or a targeted probe, not by misreading variance as signal.

```python
import hashlib
import math
from dataclasses import dataclass


class LitmusIncomparableError(RuntimeError):
    """Raised when a candidate run cannot be validly compared to its baseline."""


class LitmusNoiseFloorError(RuntimeError):
    """Raised when a delta is inside the instrument's noise floor and must not gate."""


@dataclass(frozen=True)
class RunManifest:
    suite_hash: str
    seed_list_hash: str
    temperature: float
    top_p: float
    tokenizer_id: str
    rubric_version: str


@dataclass(frozen=True)
class SuiteResult:
    manifest: RunManifest
    n_cases: int
    n_pass: int

    @property
    def pass_rate(self) -> float:
        return self.n_pass / self.n_cases


def _require_comparable(baseline: SuiteResult, candidate: SuiteResult) -> None:
    b, c = baseline.manifest, candidate.manifest
    for field in ("suite_hash", "seed_list_hash", "temperature", "top_p",
                  "tokenizer_id", "rubric_version"):
        if getattr(b, field) != getattr(c, field):
            raise LitmusIncomparableError(
                f"manifest field {field!r} differs: baseline={getattr(b, field)!r} "
                f"candidate={getattr(c, field)!r}; re-baseline before gating"
            )


def noise_floor_half_width(baseline: SuiteResult, z: float = 1.96) -> float:
    p = baseline.pass_rate
    return z * math.sqrt(2.0 * p * (1.0 - p) / baseline.n_cases)


def gate(baseline: SuiteResult, candidate: SuiteResult, z: float = 1.96) -> str:
    _require_comparable(baseline, candidate)
    delta = candidate.pass_rate - baseline.pass_rate
    e = noise_floor_half_width(baseline, z=z)
    if abs(delta) <= e:
        raise LitmusNoiseFloorError(
            f"delta={delta:+.4f} within noise floor +/-{e:.4f} at n={baseline.n_cases}; "
            "do not gate, do not open incident"
        )
    if delta < 0:
        return f"REGRESSION delta={delta:+.4f} exceeds noise floor +/-{e:.4f}; gate release"
    return f"IMPROVEMENT delta={delta:+.4f} exceeds noise floor +/-{e:.4f}; re-baseline"


def manifest_hash(suite: str, seeds: list[int], temperature: float,
                  top_p: float, tokenizer_id: str, rubric_version: str) -> RunManifest:
    return RunManifest(
        suite_hash=hashlib.sha256(suite.encode()).hexdigest()[:16],
        seed_list_hash=hashlib.sha256(",".join(map(str, seeds)).encode()).hexdigest()[:16],
        temperature=temperature,
        top_p=top_p,
        tokenizer_id=tokenizer_id,
        rubric_version=rubric_version,
    )
```