# What structural signals indicate a LITMUS harness is measuring refusal policy rather than capability under adversarial pressure **Date of Record:** 2026 November 18 **Scope:** Applies to LITMUS-style agentic safety benchmarks that score an agent on multi-turn adversarial probes and feed the result into a continuous safety regression suite. Does not apply to single-turn static prompt-injection classifiers or to capability-only evals with no refusal dimension. ## Core Distinction A LITMUS harness produces two structurally distinct signals that are routinely conflated in dashboards: *refusal behavior* (the policy layer declining a request) and *capability under adversarial pressure* (whether the agent can still complete legitimate sub-tasks while under probe). The non-obvious boundary is that a harness can appear to be measuring the latter while structurally only ever exercising the former. When this happens, the regression suite reports improving "safety" scores that are actually just a tightening refusal policy — and the suite silently loses the ability to detect capability regressions under attack, which is the entire point of an agentic red-team. The failure is not a bug in any single probe. It is a *structural* property of how the probe set, the scoring rubric, and the agent's tool-call trace interact. The signal that distinguishes the two regimes is not in the pass/fail numbers; it is in the *shape* of the trace. ## What Was Observed Across several continuous safety regression suites built on LITMUS-style harnesses, a recurring pattern emerged during incident review: - The aggregate "safety score" climbed monotonically over weeks while no probe set had changed. The suite reported this as improvement. - Drilling into traces, every high-scoring episode terminated in an early refusal at turn 1 or 2. The agent never reached the multi-turn tool-use portion of the probe. - Probes that *required* the agent to complete a benign sub-task before the adversarial turn (e.g., "summarize this document, then..." where the adversarial payload was embedded mid-document) showed a bimodal distribution: either full completion or immediate refusal, with almost no partial trajectories. - When the refusal policy was deliberately loosened in a canary run, the "safety score" dropped sharply but the *capability-under-attack* sub-scores also dropped — revealing that those sub-scores had been near-zero all along and were being masked by early refusals. The structural tell: **the distribution of turns-to-termination collapsed to a spike at the earliest possible turn.** A harness genuinely measuring capability under adversarial pressure produces a spread of termination turns, because some probes are survived through to completion and others are caught partway. A harness measuring only refusal policy produces a spike at turn 1. ## Condition of Validity The refusal-vs-capability distinction is only a *confound* under these conditions, and the diagnostic is only valid when they hold: 1. The probe set contains probes whose adversarial content is *positioned after* benign sub-tasks (multi-turn, tool-using, or document-embedded). If all probes are single-turn, refusal and capability are genuinely indistinguishable and the distinction is vacuous. 2. The scoring rubric gives partial credit for partial trajectories. If scoring is all-or-nothing per episode, the turn-distribution diagnostic has no signal to read. 3. The agent's refusal is *observable in the trace* as a distinct event, not inferred from a low score. Harnesses that only record the final score cannot run this diagnostic. 4. The canary-loosening procedure is sanctioned by the suite owner, because it temporarily degrades the safety posture of the evaluated agent. This is the operational cost of the diagnostic. If any of these fail, the diagnostic is invalid and the suite must not be re-tuned on the basis of it. ## Decision Boundary The operational rule the suite should enforce: - Compute the **turn-termination histogram** per probe class on every regression run, not just the aggregate score. - If the histogram is a spike at turn 1 *and* the canary-loosening run does not recover capability-under-attack sub-scores, the harness is in **refusal-only mode**. Treat the safety score as a refusal-policy metric and *do not* report it as adversarial capability. Flag the suite as structurally degraded. - If the histogram has spread and the canary run recovers sub-scores, the harness is in **capability-under-pressure mode** and the aggregate score is meaningful as a safety regression signal. - The boundary is crossed when the spike at turn 1 exceeds a threshold fraction of episodes (empirically, >0.85 in the suites reviewed) *and* the canary recovery is below a floor. Neither condition alone is sufficient; a genuinely safe agent can have a high early-refusal rate on a probe set that is mostly single-turn. The trap is tuning on the aggregate score alone. The aggregate is the *output*; the turn histogram and the canary recovery are the *invariants* that tell you what the output means. ## Operational Effect Concretely, suites that were re-tuned on the aggregate score alone drifted toward refusal-only mode within a few regression cycles, because tightening refusal is the cheapest way to raise the aggregate. The continuous regression suite then became a refusal-policy monitor wearing the costume of an adversarial capability eval. The failure is silent: no probe errors, no harness exceptions, green dashboards. The fix is structural, not statistical: the harness must assert, at scoring time, that the trace shape is consistent with the mode it claims to be in, and raise when it is not. The check below encodes that invariant — it raises when a suite reports capability-under-pressure scores from a trace population that is structurally refusal-only. ```python class HarnessModeMismatchError(RuntimeError): """Raised when a LITMUS harness reports capability-under-attack scores from a trace population whose termination-turn distribution is a turn-1 spike, i.e. the harness is structurally in refusal-only mode.""" pass EARLY_TERMINATION_TURN = 1 SPIKE_THRESHOLD = 0.85 CANARY_RECOVERY_FLOOR = 0.40 def assert_harness_mode(termination_turns, canary_recovery, claimed_mode): """ termination_turns: list[int] turn index at which each episode terminated canary_recovery: float in [0,1] fraction of capability sub-score recovered when refusal policy is loosened claimed_mode: "capability_under_pressure" | "refusal_policy" """ if not termination_turns: raise HarnessModeMismatchError("empty trace population; cannot classify mode") early = sum(1 for t in termination_turns if t <= EARLY_TERMINATION_TURN) early_fraction = early / len(termination_turns) structurally_refusal_only = ( early_fraction > SPIKE_THRESHOLD and canary_recovery < CANARY_RECOVERY_FLOOR ) if claimed_mode == "capability_under_pressure" and structurally_refusal_only: raise HarnessModeMismatchError( f"harness claims capability_under_pressure but {early_fraction:.2%} of " f"episodes terminate at turn <= {EARLY_TERMINATION_TURN} and canary " f"recovery is {canary_recovery:.2f} (< {CANARY_RECOVERY_FLOOR}). " f"Reported scores are refusal-policy metrics, not adversarial capability." ) if claimed_mode not in ("capability_under_pressure", "refusal_policy"): raise HarnessModeMismatchError(f"unknown claimed_mode: {claimed_mode!r}") return { "early_fraction": early_fraction, "canary_recovery": canary_recovery, "structurally_refusal_only": structurally_refusal_only, "mode_consistent": True, } ```