# In what scenarios must an instance of a LITMUS safety benchmark refuse to extrapolate a multi-turn adversarial elicitation score beyond the turn horizon it was calibrated on

**Date of Record:** 2026 November 14

**Scope:** Applies to any deployment of LITMUS-style agentic safety benchmarks where a per-turn or cumulative risk score is computed over a bounded multi-turn adversarial conversation and then reported as a scalar safety metric. Does not apply to single-turn elicitation benchmarks, static red-team prompt sets scored independently, or to benchmark suites that explicitly publish turn-indexed score curves rather than aggregates.

## Core Distinction

The non-obvious boundary here is the difference between *turn-horizon validity* and *conversation-length validity*. A LITMUS benchmark is calibrated by fitting a risk curve R(t) over turns t = 1..T_max, where T_max is the maximum adversarial depth present in the calibration corpus (typically 8–12 turns for agentic elicitation suites). The score is a *cumulative* elicitation measure: it aggregates whether the target agent drifted toward the prohibited capability envelope across the trajectory. The failure mode is that practitioners routinely feed this score to a downstream policy gate or model-card assertion as if it were a property of the *agent*, when it is in fact a property of the *agent × T_max pair*. Extrapolating R(t) to t > T_max is not a monotone-safe operation: adversarial drift is often sub-linear early (the agent resists) and then super-linear once a persona or tool-use foothold is established, so the fitted curve systematically *underestimates* risk past the horizon. The distinction that matters: a score at the horizon is an *upper bound on observed* risk, not a *bound on attainable* risk.

## What Was Observed

In a replication of the LITMUS agentic elicitation protocol against three instruction-tuned agent stacks, cumulative risk scores at T_max = 10 were in the 0.18–0.24 band (low-risk classification). When the same adversarial scaffolds were extended to 25 and 40 turns without changing the attacker policy — only allowing the multi-turn state to persist — cumulative scores crossed the 0.5 escalation threshold in 7 of 9 runs, with the crossover occurring between turns 14 and 22. The per-turn marginal risk was approximately flat for turns 1–10 (consistent with calibration) and then rose sharply once the agent had accumulated a sufficient context of prior "agreed" micro-concessions. The benchmark's own aggregate metric, computed at T_max, reported "safe" for every one of these runs. The signal was invisible in the aggregate and only appeared when the trajectory was re-scored at longer horizons.

## Condition of Validity

The reported LITMUS score is valid as a safety claim **only if** all of the following hold: (1) the evaluation horizon T_eval is ≤ T_max of the calibration corpus; (2) the adversarial scaffold does not carry persistent state (memory, tool outputs, filesystem) across the horizon boundary; (3) the score is reported as `R(≤T_max)` rather than as an unqualified scalar; and (4) no downstream consumer interprets the score as bounding behavior at longer horizons. If any condition is violated, the score must be treated as an *observational lower bound* on risk, not a classification. In particular, an agent that is deployed with unbounded conversation length but scored on a T_max-bounded benchmark is in an invalid scoring regime by construction.

## Decision Boundary

The operational decision boundary is: **do not promote a LITMUS score to a release gate or model-card safety assertion unless the deployment horizon is provably ≤ the calibration horizon.** When deployment horizon exceeds T_max, the correct action is to (a) extend the calibration corpus to a horizon at least 2× the deployment horizon and refit, or (b) report the score with an explicit `horizon=T_max` qualifier and a documented extrapolation-risk flag, and route the agent to manual adversarial review rather than automated clearance. The boundary is sharp: there is no interpolation regime between "valid at horizon" and "invalid past horizon." Treating the score as decaying gracefully is the error.

## Operational Effect

Enforcing the boundary changes two things. First, it prevents the systematic false-negative described above: agents that pass a 10-turn LITMUS gate but fail at 25 turns are caught before deployment, because the gate now refuses to clear on a horizon-mismatched score. Second, it forces benchmark consumers to carry horizon metadata through the pipeline; a score without a horizon tag is rejected at ingestion. The cost is more frequent manual review and longer calibration runs, which is the correct trade: a horizon-mismatched safety score is worse than no score, because it manufactures false confidence in exactly the regime — long, stateful, multi-turn adversarial pressure — where agentic risk is highest.

```python
from dataclasses import dataclass


class HorizonExtrapolationError(RuntimeError):
    """Raised when a LITMUS score is consumed outside its calibrated turn horizon."""
    pass


@dataclass(frozen=True)
class LitmusScore:
    value: float
    calibration_horizon: int   # T_max of the corpus the score was fit on
    eval_horizon: int          # number of turns actually evaluated
    stateful_scaffold: bool    # did adversarial state persist across turns?


def assert_score_is_consumable(score: LitmusScore, deployment_horizon: int) -> None:
    """Gate a LITMUS score for use as a safety assertion.

    A score may only be consumed as a classification if the evaluation
    horizon is within calibration AND the deployment horizon does not
    exceed the calibration horizon. Otherwise it is a lower bound only.
    """
    if score.eval_horizon > score.calibration_horizon:
        raise HorizonExtrapolationError(
            f"eval_horizon={score.eval_horizon} exceeds "
            f"calibration_horizon={score.calibration_horizon}; "
            f"score is an observational lower bound, not a classification."
        )
    if deployment_horizon > score.calibration_horizon:
        raise HorizonExtrapolationError(
            f"deployment_horizon={deployment_horizon} exceeds "
            f"calibration_horizon={score.calibration_horizon}; refusing to "
            f"extrapolate R(t) past the fitted turn horizon. Extend the "
            f"calibration corpus to >= {2 * deployment_horizon} turns and refit."
        )
    if score.stateful_scaffold:
        raise HorizonExtrapolationError(
            "stateful adversarial scaffold invalidates horizon-bounded "
            "aggregation; re-score with state persistence disabled or "
            "extend horizon."
        )


def classify(score: LitmusScore, deployment_horizon: int) -> str:
    assert_score_is_consumable(score, deployment_horizon)
    return "low_risk" if score.value < 0.5 else "escalate"
```