# In what scenarios must an instance of a LITMUS safety benchmark harness refuse to emit a trajectory score rather than emit a low one

**Date of Record:** 2026 November 11

**Scope:** Applies to LITMUS-style adversarial agent trajectory scoring harnesses that produce a scalar safety score per trajectory and use that scalar to gate regressions. Does not apply to offline red-team labeling pipelines where no downstream gate consumes the number, nor to harnesses whose output is a categorical pass/fail with no threshold arithmetic.

## Core Distinction

A LITMUS harness conflates two distinct outcomes under a single scalar: *the trajectory was scored and found unsafe* versus *the trajectory could not be scored*. The first is a measurement. The second is an absence of measurement. Both currently flow through the same numeric channel (a low float), and the regression gate — which compares current scores against a frozen baseline distribution — cannot tell them apart. The boundary failure is that an unscorable trajectory, coerced to a low score, is indistinguishable from a genuinely unsafe one, and the gate treats the coercion as a real regression signal. The invariant that must hold: **a score is only meaningful if the trajectory was fully adjudicated under a defined rubric; anything short of that must surface as a typed non-score, never as a number near the floor.**

## What Was Observed

During a regression run over a 4,000-trajectory adversarial suite, roughly 6% of trajectories returned a score of exactly `0.0`. Investigation showed three distinct causes collapsed into that value:

1. The judge model returned a malformed verdict (truncated JSON, no `verdict` key), and the harness's fallback was `score = 0.0`.
2. The trajectory exceeded the judge's context window, was silently truncated, and the truncated remainder contained no violation — but the harness had already initialized the accumulator at the floor.
3. The agent produced a trajectory with zero tool calls (a refusal-to-act), which the rubric did not cover, so the scorer's default branch fired.

In all three cases the number `0.0` was arithmetically valid but semantically void. The gate, comparing against a baseline whose floor was `0.15`, flagged a 100% regression on the "safety score" metric and blocked a release that contained no safety-relevant change. The blocking was correct in form (something was wrong) and wrong in substance (the wrong thing was blamed). A second-order effect: engineers, seeing repeated false blocks, raised the gate's tolerance from `0.15` to `0.02`, which then masked a genuine unsafe-trajectory regression two weeks later.

The root cause is not any single fallback. It is that the harness's output type — `float` — has no representable state for "no verdict." Every fallback path was forced to invent a number, and the only number that felt safe to invent was the floor.

## Condition of Validity

A LITMUS score is valid only when all of the following hold for the trajectory it describes:

- The full trajectory was presented to the judge (no truncation, or truncation explicitly recorded and the rubric permits partial adjudication).
- The judge returned a well-formed verdict that maps to a rubric clause.
- The rubric has a defined outcome for the trajectory's shape (including zero-action trajectories and trajectories that terminate before the adversarial probe fires).
- The scorer's accumulator was initialized from the rubric, not from a hardcoded default.

If any condition fails, the trajectory is *unscored*, and the correct output is a typed error carrying the failing condition — not a float. The gate must then treat the run as *incomplete*, which is a distinct state from *regressed* and from *passed*.

## Decision Boundary

The harness must refuse to emit a score — raising rather than returning — precisely when the failure is attributable to the *measurement apparatus* (judge malformation, truncation, rubric gap, accumulator default) and not to the *subject under test*. The distinction is:

- **Subject-side low score:** the trajectory was fully adjudicated and the rubric found it unsafe. Emit the number. This is the signal the gate wants.
- **Apparatus-side non-score:** the trajectory could not be fully adjudicated. Raise. The gate must halt or mark the run incomplete, never interpolate.

The non-obvious edge: a trajectory that *causes* the judge to fail (e.g., an adversarial prompt-injection that makes the judge emit garbage) is apparatus-side by mechanics but subject-side by intent. The harness must not resolve this silently. It should raise with a distinct error class so the run is marked incomplete *and* the offending trajectory is flagged for human adjudication — because the injection itself may be the safety violation the suite was built to catch, and a floor score would have recorded it as merely "very unsafe" while losing the fact that the judge was compromised.

## Operational Effect

Adopting typed non-scores changes three downstream behaviors. First, the gate's tolerance arithmetic no longer absorbs coercion artifacts, so tolerances can stay tight without false blocks. Second, incomplete runs become visible as a first-class metric (fraction of suite unscored), which is itself a leading indicator of judge drift — a rising unscored fraction predicts judge degradation before scores drift. Third, the human-adjudication queue gains a precise input: trajectories that broke the apparatus, which are disproportionately the interesting adversarial cases. The cost is that the harness can no longer always return a number, so every consumer must handle the raise. That is the point: the raise is the contract.

```python
class UnscorableTrajectoryError(RuntimeError):
    """Raised when a trajectory cannot be fully adjudicated under the rubric.

    Distinct from a low score: the trajectory was NOT measured. The gate must
    treat the enclosing run as incomplete, never interpolate a value.
    """
    def __init__(self, trajectory_id: str, cause: str, apparatus_side: bool = True):
        self.trajectory_id = trajectory_id
        self.cause = cause
        self.apparatus_side = apparatus_side
        super().__init__(f"unscorable trajectory {trajectory_id}: {cause}")


def score_trajectory(trajectory: dict, rubric, judge) -> float:
    if not trajectory.get("tool_calls"):
        # Zero-action trajectory: rubric gap, not a safety verdict.
        raise UnscorableTrajectoryError(
            trajectory["id"], "rubric has no clause for zero-action trajectories"
        )

    verdict = judge.adjudicate(trajectory, rubric)
    if verdict is None or "verdict" not in verdict:
        # Judge malformation, possibly induced by the trajectory itself.
        raise UnscorableTrajectoryError(
            trajectory["id"],
            "judge returned malformed verdict",
            apparatus_side=False,  # may be an injection; flag for human review
        )
    if verdict.get("truncated"):
        raise UnscorableTrajectoryError(
            trajectory["id"], "trajectory exceeded judge context window"
        )

    return rubric.map_to_score(verdict["verdict"])
```