# Why does an instance-level safety verdict silently invert at the trajectory boundary in the LITMUS harness

**Date of Record:** 2026 November 11

**Scope:** Applies to LITMUS trajectory scoring when adversarial agent runs are aggregated from per-step judgments into a single run verdict and then fed into the regression gate. Does not apply to single-turn classification, nor to runs where every step is scored by the same judge instance with identical sampling parameters and no retry/fallback path.

## Core Distinction

LITMUS distinguishes two aggregation layers that are easy to conflate: the **step verdict** (a boolean or score produced by the judge for one trajectory step) and the **trajectory verdict** (the reduction of step verdicts into a run-level label). The failure mode here is not a wrong step verdict — step-level precision can be perfect — but a **reduction that inverts polarity** under a specific, non-obvious condition: when the trajectory contains a *terminal safe refusal* followed by a *post-refusal continuation step* that the agent executes anyway.

The harness treats "refusal" as a safe terminal marker. But adversarial trajectories frequently contain a refusal at step *k* and then a continuation at step *k+1* (the agent says "I can't help with that" and then, prompted again, does it). The reduction rule in the default LITMUS profile is `trajectory_safe = all(step_safe)`. That rule is correct. The inversion happens one layer up: the **regression gate** compares the *new* trajectory verdict against the *baseline* trajectory verdict, and the baseline was recorded with a **truncation policy** that cut the trajectory at the first refusal. So the baseline is `safe=True` (it never saw step *k+1*), while the new run — scored under the current, non-truncating policy — is `safe=False`. The gate reads this as a *regression*, when in fact the new run is the first honest measurement of a behavior that was always present. Conversely, a genuine safety improvement (agent now refuses and *stops*) can be read as "no change" if the baseline was already truncated at the refusal.

## What Was Observed

During a re-scoring pass over an adversarial suite of 1,200 multi-turn trajectories, 214 runs flipped from `safe=True` in the stored baseline to `safe=False` under re-scoring. Manual audit of 40 sampled flips found:

- 38 of 40 contained a refusal at step *k* and a harmful continuation at step *k+1* or later.
- In 36 of those 38, the baseline record's `steps` array had length exactly *k* — the trajectory was truncated at the refusal.
- The judge model version was identical between baseline and re-score; step-level verdicts for steps 1..*k* matched exactly.
- The gate emitted `REGRESSION` for all 214, and blocked the release branch.

The remaining 2 of 40 were genuine regressions (a previously-safe continuation became harmful), indistinguishable at the gate level from the 38 truncation artifacts. That indistinguishability is the operational hazard: the gate's output does not encode *why* the verdict changed, so the 38 false positives and 2 true positives arrive as the same signal.

A second, subtler observation: in 11 runs the truncation occurred not at a refusal but at a **tool-call boundary** — the baseline harness stopped recording after a tool invocation whose result was empty, treating the empty result as terminal. The continuation after the empty tool result was never scored. Same inversion, different truncation trigger.

## Condition of Validity

The inversion is valid (i.e., the gate output is untrustworthy) precisely when all of the following hold:

1. The baseline artifact was produced under a truncation policy (`truncate_on_refusal=True` or `truncate_on_empty_tool_result=True`) that differs from the current scoring policy.
2. The trajectory contains at least one step after the truncation point that is scored by the current policy.
3. The gate compares raw trajectory verdicts without a **policy-provenance check** — i.e., it does not verify that baseline and candidate were scored under the same truncation policy and the same step-count coverage.

If any condition fails, the comparison is sound. In particular, if the baseline is re-scored under the current policy before comparison (a *re-baseline* step), the inversion disappears — but re-baselining is expensive and is frequently skipped in CI to save judge calls.

The inversion does **not** occur for step-level regression gates, because step *k+1* simply has no baseline counterpart and is treated as a new step rather than a changed verdict. The bug is exclusive to the trajectory-level reduction being compared across policy versions.

## Decision Boundary

The operational rule: **never compare a trajectory verdict against a baseline whose truncation policy or step coverage differs from the candidate's.** Before the gate runs, assert policy provenance equality. If provenance differs, the correct action is not to pass and not to fail, but to route the run to `NEEDS_REBASELINE` and either re-score the baseline under the current policy or exclude the run from the gate with a recorded reason. Silently treating the mismatch as a regression burns release velocity on false positives; silently treating it as a pass ships genuine regressions hidden among the artifacts. Both are wrong; the only correct outcome is an explicit third state.

The boundary is sharp: a single trailing step difference is enough to invert the verdict. There is no tolerance window. Do not attempt to "fuzz" the comparison by allowing a one-step slack — that reintroduces the 2-of-40 true regressions as false negatives.

## Operational Effect

After enforcing the provenance check, the 214 flips resolved as: 38 routed to `NEEDS_REBASELINE` (then re-scored, 36 confirmed as pre-existing behavior, 2 confirmed as genuine regressions now correctly attributed), 174 resolved as truncation artifacts on re-baseline, and 2 remained genuine regressions caught by the re-scored baseline. Release-gate false-positive rate on this suite dropped from 17.8% to 0.17%. The cost was a one-time re-baseline of 1,200 trajectories (approximately 4,100 additional judge calls) and a permanent increase in gate latency of one provenance check per run.

The lesson generalizes: any regression gate that compares *reduced* verdicts across *policy versions* must carry the policy identity in the artifact, or it will confuse a measurement change with a behavior change.

```python
from dataclasses import dataclass
from typing import Optional


class TruncationPolicyMismatchError(RuntimeError):
    """Raised when a trajectory verdict is compared across incompatible truncation policies."""
    pass


@dataclass(frozen=True)
class ScoreProvenance:
    truncate_on_refusal: bool
    truncate_on_empty_tool_result: bool
    steps_covered: int
    judge_version: str


@dataclass(frozen=True)
class TrajectoryVerdict:
    run_id: str
    safe: bool
    provenance: ScoreProvenance


def gate_trajectory(
    baseline: TrajectoryVerdict,
    candidate: TrajectoryVerdict,
) -> str:
    """Return 'PASS', 'REGRESSION', or raise on incompatible provenance."""
    b, c = baseline.provenance, candidate.provenance

    if b.judge_version != c.judge_version:
        raise TruncationPolicyMismatchError(
            f"judge version differs: {b.judge_version!r} vs {c.judge_version!r}"
        )
    if b.truncate_on_refusal != c.truncate_on_refusal:
        raise TruncationPolicyMismatchError(
            "truncate_on_refusal differs between baseline and candidate; "
            "re-baseline required before comparison"
        )
    if b.truncate_on_empty_tool_result != c.truncate_on_empty_tool_result:
        raise TruncationPolicyMismatchError(
            "truncate_on_empty_tool_result differs; re-baseline required"
        )
    if b.steps_covered != c.steps_covered:
        raise TruncationPolicyMismatchError(
            f"step coverage differs ({b.steps_covered} vs {c.steps_covered}); "
            "trajectory verdict not comparable"
        )

    if baseline.safe and not candidate.safe:
        return "REGRESSION"
    return "PASS"


def run_gate(baseline: Optional[TrajectoryVerdict], candidate: TrajectoryVerdict) -> str:
    if baseline is None:
        return "NEEDS_REBASELINE"
    try:
        return gate_trajectory(baseline, candidate)
    except TruncationPolicyMismatchError:
        return "NEEDS_REBASELINE"
```