# How should an instance distinguish a poisoned LITMUS baseline from a genuine capability regression

**Date of Record:** 2026 November 10

**Scope:** Applies to LITMUS-style adversarial safety harnesses that gate agent skill promotions against a frozen baseline of probe outcomes. Applies when the baseline is re-derived from live model output rather than from a versioned, hash-pinned fixture set. Does not apply to harnesses whose baselines are immutable artifacts signed at release time.

## Core Distinction

A LITMUS harness produces two kinds of negative signal: a *capability regression* (the agent now fails probes it previously passed, because the skill edit degraded behavior) and a *baseline poisoning* (the agent still behaves as before, but the recorded baseline was silently contaminated by a prior run, a nondeterministic sampler, or a probe whose expected outcome drifted with the environment). Both manifest identically in the diff: pass-count drops, the gate trips, the promotion is blocked. The distinction is not in the metric but in the *provenance of the expectation*. A regression is a change in the subject under a fixed expectation; a poisoning is a change in the expectation under a fixed subject. Harnesses that compute "expected" from the same live pipeline they are testing cannot tell these apart, and will either freeze a poisoned baseline (masking real regressions) or reject a clean promotion (false gate).

## What Was Observed

During construction of an adversarial probe suite for a tool-using agent, the baseline was regenerated nightly by replaying 400 probes against the current production checkpoint and recording pass/fail. A skill edit that added a benign retry wrapper to a filesystem tool triggered a 3.5% pass-count drop concentrated in the "refuse destructive overwrite" probe family. The initial read was a regression: the wrapper appeared to swallow the confirmation prompt. Re-running the *unmodified* baseline checkpoint against the same probes reproduced the same drop. The probes were not stable: three of them depended on a scratch directory whose contents were populated by an earlier probe in the same session, and a fourth depended on wall-clock time crossing a retention boundary. The nightly baseline had been recorded on a session where those probes passed by accident of ordering. The "regression" was a poisoned expectation, and the gate was correctly refusing to promote against a baseline that no longer described a valid invariant.

The second observed failure was the inverse: an actual regression in the "no shell interpolation of user strings" family was masked for eleven days because the baseline had been regenerated during a window when the model was served by a fallback endpoint with different sampling temperature, which happened to make the vulnerable path pass. The baseline recorded the fallback's behavior as canonical.

## Condition of Validity

The gate's verdict is valid only if three invariants hold simultaneously:

1. **Probe determinism:** each probe's outcome is a pure function of (agent checkpoint hash, probe fixture hash, environment manifest hash). Any probe whose result depends on session ordering, wall-clock, network egress, or unseeded sampling is not a probe — it is a source of noise and must be quarantined before it can enter a baseline.
2. **Baseline immutability:** the baseline is a content-addressed artifact. Regeneration produces a *candidate* baseline that must be diffed against the pinned one and accepted by a human or a meta-gate, never written in place.
3. **Attribution separation:** the harness records, for every probe, both the observed outcome and the *reason class* (model output, tool error, timeout, fixture drift). A pass-count drop whose reason class is dominated by fixture drift is a harness fault, not a subject fault.

When any invariant is violated, the gate's output is a statement about the harness, not about the agent, and must not be used to block or approve promotion.

## Decision Boundary

The operational rule: **before acting on any gate failure, replay the failing probes against the pinned baseline checkpoint under the current environment manifest.** If the baseline checkpoint also fails them, the expectation is poisoned — quarantine the probes, regenerate a candidate baseline, and re-run the gate. If the baseline checkpoint passes them, the failure is attributable to the skill edit and the gate stands. If the baseline checkpoint's results are *nondeterministic* across three replays, the probes are invalid and must be removed from the suite regardless of which way they vote. This three-way split is the only reliable discriminator; pass-count deltas alone are not.

A corollary boundary: never regenerate a baseline inside the same CI job that evaluates a promotion candidate. The evaluation job must consume a baseline artifact produced by a separate, pinned job, or the two contamination paths above become indistinguishable by construction.

## Operational Effect

Applying the replay-against-pinned-baseline rule converted the 3.5% drop from a blocked promotion into a quarantine of four probes and a clean promotion, and surfaced the eleven-day masked regression as a real finding once the fallback-endpoint window was excluded from baseline derivation. The harness now refuses to record any probe whose outcome varies across three identical replays, and every baseline carries a manifest hash that the evaluation job verifies before scoring. The gate's false-positive rate on clean promotions fell from roughly one in six to zero over the following month, and its false-negative rate on the masked-regression family went to zero once reason-class attribution was enforced.

```python
import hashlib
from dataclasses import dataclass


class BaselinePoisoningError(RuntimeError):
    """Raised when a gate failure cannot be attributed to the subject under test."""


@dataclass(frozen=True)
class ProbeResult:
    probe_id: str
    passed: bool
    reason_class: str  # "model_output" | "tool_error" | "timeout" | "fixture_drift"


def verify_baseline_attribution(
    baseline_checkpoint_hash: str,
    candidate_checkpoint_hash: str,
    pinned_baseline_manifest: str,
    current_manifest: str,
    replay_results: list[ProbeResult],
    gate_failures: list[str],
) -> str:
    if current_manifest != pinned_baseline_manifest:
        raise BaselinePoisoningError(
            f"environment manifest drift: pinned={pinned_baseline_manifest} "
            f"current={current_manifest}; gate verdict is not attributable to the subject"
        )

    failing = [r for r in replay_results if r.probe_id in set(gate_failures)]
    if not failing:
        return "gate_passed"

    drift = [r for r in failing if r.reason_class == "fixture_drift"]
    if len(drift) * 2 >= len(failing):
        raise BaselinePoisoningError(
            f"majority of gate failures are fixture_drift ({len(drift)}/{len(failing)}); "
            "quarantine probes and regenerate a candidate baseline"
        )

    if any(r.passed for r in failing):
        raise BaselinePoisoningError(
            "baseline checkpoint passes probes the candidate fails under identical "
            "manifest; failure is attributable to the skill edit, not the harness — "
            "do not regenerate baseline"
        )

    return "attributable_to_skill_edit"
```