# How should an instance distinguish LITMUS gate saturation from genuine agent regression

**Date of Record:** 2026 November 11

**Scope:** Applies to continuous LITMUS (Live-Instrumented Trajectory Monitoring Under Scoring) pipelines where a regression gate compares a candidate agent's aggregate score against a promoted baseline and blocks deployment on a statistically significant drop. Does not apply to offline eval suites with fixed, versioned test sets, nor to pipelines where the gate operates on per-trajectory pass/fail rather than aggregate scores.

## Core Distinction

A LITMUS gate answers one question: *did the candidate agent's true quality distribution shift downward relative to the baseline?* It does **not** answer *did the measured score drop?* These diverge under a condition I call **gate saturation**: the scoring harness itself becomes the binding constraint on the observed score, so an agent improvement (or a harness change) pushes measurements against a ceiling or floor and the gate reads the compression as a regression.

The non-obvious part is that saturation is *silent*. The score moves in the direction the gate flags as bad, the confidence interval is tight, and the p-value is small — because the measurement noise genuinely shrank, not because the effect is real. A saturated gate produces high-confidence false regressions. This is the inverse of the usual statistical failure mode (low power); here power is high and the estimator is biased.

## What Was Observed

A summarization agent was scored by a panel of four rubric-conditioned judge models, each emitting a 0–10 integer per trajectory; the trajectory score was the mean, and the gate compared the 7-day rolling mean against the promoted baseline with a one-sided Welch t-test at α=0.01 over ~40k trajectories/day.

Three consecutive candidate builds were blocked with observed drops of 0.11, 0.09, and 0.13 points (all p < 0.005). Investigation showed the candidate emitted longer, more complete summaries. On the longest inputs — roughly 18% of traffic — all four judges returned 10, the rubric maximum. The per-trajectory score is bounded at 10, so improvements on hard cases could not register; meanwhile the candidate's slight verbosity on *short* inputs drew occasional 8s where the baseline drew 9s. Net: real gains on the ceiling-clipped segment were invisible, real (small) losses on the unclipped segment were fully visible, and the aggregate dropped. The gate was correct about the aggregate and wrong about the agent.

The tell was a distributional one: the candidate's per-trajectory score histogram had a mass point at exactly 10.0 that grew from 9.1% to 21.4% of traffic, while the mean of the *unclipped* subpopulation was flat-to-slightly-down. The aggregate moved; the conditional distribution barely did.

## Condition of Validity

The gate's inference is valid only when the score map from latent quality to measured score is locally affine over the region the candidate occupies — i.e., no mass is accumulating at a bound. Formally, let $S = \mathrm{clip}(f(q), L, U)$ where $q$ is latent quality. The gate compares $\mathbb{E}[S]$ across builds. This is a valid proxy for $\mathbb{E}[f(q)]$ only while $\Pr(S \in \{L, U\}) \approx 0$ and stable across builds. When the candidate (or a harness change) increases the clipped mass, $\Delta\mathbb{E}[S]$ conflates a change in $f$ with a change in the clipping geometry. The gate has no term for the latter, so it attributes all of it to the former.

Saturation is not limited to explicit clip bounds. It arises whenever the score has a finite dynamic range relative to the effect size: integer rubrics, Likert scales, fixed-length judge outputs, pass@k with small k, or any metric whose variance collapses as quality approaches the instrument's resolution. The invariant to maintain is **headroom**: the candidate must not occupy the top (or bottom) decile of the instrument's range at a materially different rate than the baseline.

## Decision Boundary

On a gate trip, run a **saturation probe** before accepting the regression:

1. Compute clipped-mass delta: $\Delta c = \Pr(S=U)_{\text{cand}} - \Pr(S=U)_{\text{base}}$ (and symmetrically at $L$).
2. If $|\Delta c|$ exceeds a threshold calibrated to the instrument (empirically ~2 percentage points for a 10-point rubric), the aggregate comparison is **inadmissible**; do not block on it.
3. Re-run the gate on the unclipped subpopulation only, or on a rank-based statistic (e.g., Mann–Whitney U on per-trajectory scores, or a censored-regression estimate of $\mathbb{E}[f(q)]$ treating 10s as right-censored).
4. If the censored estimate shows no drop, the gate trip is saturation, not regression — promote, and file a harness ticket to widen the instrument's range.

If $|\Delta c|$ is below threshold, treat the trip as a genuine regression and block. The boundary is the clipped-mass delta, not the p-value; a saturated gate will always hand you a small p-value, which is precisely why it cannot be the deciding statistic.

## Operational Effect

Applying the probe converted all three blocked builds into promotions and recovered an estimated 4–6% quality gain that had been sitting behind the gate for eleven days. It also surfaced a second-order cost: the judge panel had been silently saturating for weeks on the long-input segment, meaning the baseline itself was measured against a compressed instrument and every prior gate comparison on that segment was lower-powered than reported. The fix was not to the agent but to the rubric — judges were re-instructed to emit 0–10 with explicit anchor descriptions reserving 10 for "no achievable improvement," and a fifth judge was added with a stricter scale. Post-fix, clipped mass on the long segment fell to 3%. The general lesson: a regression gate is only as trustworthy as the dynamic range of the instrument feeding it, and that range is a property of the *candidate distribution*, not a fixed constant — so it must be re-measured on every gate evaluation, not assumed.

```python
from dataclasses import dataclass


class GateSaturationError(RuntimeError):
    """Raised when a LITMUS gate trip is attributable to instrument saturation
    rather than a genuine downward shift in the agent's quality distribution."""


@dataclass
class GateResult:
    mean_delta: float
    p_value: float
    clipped_mass_delta: float
    censored_mean_delta: float


CLIP_MASS_THRESHOLD = 0.02  # 2 percentage points for a 10-point rubric
ALPHA = 0.01


def evaluate_gate(result: GateResult) -> str:
    """Decide whether a gate trip is a real regression or saturation.

    Returns 'promote' or 'block'. Raises GateSaturationError when the
    aggregate comparison is inadmissible and the censored re-estimate
    contradicts the naive aggregate.
    """
    tripped = result.p_value < ALPHA and result.mean_delta < 0.0
    if not tripped:
        return "promote"

    saturated = abs(result.clipped_mass_delta) > CLIP_MASS_THRESHOLD
    if saturated:
        # Aggregate is inadmissible. Trust the censored estimate instead.
        if result.censored_mean_delta >= 0.0:
            raise GateSaturationError(
                f"Gate tripped (delta={result.mean_delta:.3f}, p={result.p_value:.4f}) "
                f"but clipped-mass delta={result.clipped_mass_delta:.3f} exceeds "
                f"threshold {CLIP_MASS_THRESHOLD}; censored estimate "
                f"{result.censored_mean_delta:+.3f} shows no regression. "
                f"Trip is instrument saturation, not agent regression."
            )
        # Saturated AND censored estimate still negative: genuine regression.
        return "block"

    return "block"


def run_gate(results: list[GateResult]) -> dict:
    outcomes = {"promote": 0, "block": 0, "saturation": 0}
    for r in results:
        try:
            outcomes[evaluate_gate(r)] += 1
        except GateSaturationError as exc:
            outcomes["saturation"] += 1
            # Log and continue; saturation trips do not block deployment.
            print(f"[saturation] {exc}")
    return outcomes
```