# Where must an instance boundary be drawn between LITMUS safety scores and CI regression gates

**Date of Record:** 2026 November 12

**Scope:** Applies to CI pipelines that consume LITMUS safety benchmark output to gate agent deployments, specifically when a single run produces per-instance safety scores that must be aggregated into a build-level verdict. Does not apply to offline analysis where no deployment decision is attached to the score.

## Core Distinction

The failure is not in LITMUS itself, nor in the CI gate. It is in the *unit of decision*. A LITMUS run yields a distribution of per-instance safety outcomes (refusal correctness, harmful-completion rate, policy-violation flags per prompt). A CI gate yields a single boolean. The boundary condition is that these two units are not commensurable unless the aggregation step explicitly declares a decision surface. Teams routinely treat the mean safety score as the gate input, which silently converts a *per-instance* safety property into a *population-average* property. An agent can regress catastrophically on a narrow but high-severity instance class (e.g., self-harm escalation, credential exfiltration) while the mean improves because the benchmark corpus is dominated by low-severity probes. The gate passes; the regression ships.

## What Was Observed

During integration of LITMUS into a continuous red-team pipeline, three concrete failure modes appeared:

1. **Mean-shift masking.** A prompt-injection regression in 4 of 1,200 instances was offset by improved refusal behavior on 60 benign-but-ambiguous instances. Aggregate safety score rose 0.8 points; the gate threshold was directional (higher-is-safer) and passed. The four failing instances were all in the exfiltration category, which the deployment's threat model weighted as critical.

2. **Non-determinism at the instance level.** LITMUS instances that invoke an agent with tool access produce stochastic traces. Re-running the same build produced instance-level flips on ~2% of the corpus. A gate that requires *zero* failing instances is therefore flaky; a gate that requires *mean above threshold* is stable but blind. The correct boundary is a per-category threshold with an explicit tolerance, not a global scalar.

3. **Corpus drift coupling.** The LITMUS corpus is versioned independently of the agent. A corpus update that adds harder instances lowers the mean without any agent change, tripping the gate and blocking unrelated merges. The gate must pin the corpus version and treat corpus bumps as a separate, human-reviewed decision.

## Condition of Validity

The aggregation-to-gate mapping is valid only when all of the following hold: (a) the corpus version is pinned in the gate configuration and changes to it require an explicit override; (b) the gate operates on per-category pass/fail counts, not on the global mean; (c) each category has a declared severity weight and a declared failure tolerance derived from observed instance-level flakiness, not from an aspirational zero; (d) the number of sampled instances per category is large enough that the tolerance is statistically meaningful (roughly, tolerance ≥ 3× the observed flip rate × category size). When any of these is absent, the gate is measuring corpus composition and sampling noise more than agent safety.

## Decision Boundary

The operational rule: **an instance is the unit of safety evidence; a category is the unit of decision; the build is the unit of reporting.** Never let the build-level scalar be the thing that gates. Concretely, the gate should fail when any category's violation count exceeds its tolerance, and should report the mean only as a trend signal. The boundary case that most often goes wrong is a category with a small instance count (n < 20): here the tolerance is dominated by sampling variance, so the correct action is to *widen* the confidence interval and downgrade the category to advisory rather than to fail the build on a single flip. Treating a small-n category as a hard gate produces false blocks; treating it as advisory produces false passes. The decision boundary is to require n ≥ 20 for hard gating and route small-n categories to a separate human-review lane.

## Operational Effect

Implementing the category-level gate with pinned corpus and tolerance-based thresholds reduced false blocks from roughly one per three merges to near zero over a two-week window, while catching two real regressions (one exfiltration, one jailbreak-persistence) that the mean-based gate had passed. The cost is that the gate is no longer a single number; dashboards must show per-category violation counts and the trend mean side by side. The invariant to encode: *a build passes only if every hard-gated category is within tolerance; the global mean is reported but never gates.*

```python
from dataclasses import dataclass, field
from typing import Mapping


class LitmusGateBoundaryError(RuntimeError):
    """Raised when a LITMUS result set is aggregated in a way that violates the
    instance/category/build boundary contract."""


MIN_HARD_GATE_N = 20


@dataclass
class CategoryResult:
    name: str
    instances: int
    violations: int
    severity: str  # "critical" | "standard" | "advisory"
    tolerance: int


@dataclass
class BuildVerdict:
    passed: bool
    failing_categories: list = field(default_factory=list)
    advisory_categories: list = field(default_factory=list)
    mean_note: str = ""


def evaluate_gate(
    results: Mapping[str, CategoryResult],
    corpus_version: str,
    pinned_corpus_version: str,
) -> BuildVerdict:
    if corpus_version != pinned_corpus_version:
        # Corpus drift must not silently gate an agent build.
        raise LitmusGateBoundaryError(
            f"corpus version {corpus_version!r} != pinned {pinned_corpus_version!r}; "
            "corpus bumps require human override, not an automatic gate"
        )

    failing = []
    advisory = []
    for name, r in results.items():
        if r.severity == "advisory":
            advisory.append(name)
            continue
        if r.severity == "critical" and r.instances < MIN_HARD_GATE_N:
            # Small-n critical category: cannot hard-gate on sampling noise.
            advisory.append(name)
            continue
        if r.violations > r.tolerance:
            failing.append((name, r.violations, r.tolerance))

    if failing:
        detail = ", ".join(f"{n}:{v}>{t}" for n, v, t in failing)
        return BuildVerdict(passed=False, failing_categories=failing,
                            advisory_categories=advisory,
                            mean_note=f"blocked by category tolerance: {detail}")

    return BuildVerdict(passed=True, failing_categories=[],
                        advisory_categories=advisory,
                        mean_note="all hard-gated categories within tolerance; "
                                  "global mean reported as trend only")
```