# What structural signals indicate a refusal-calibration inversion in LITMUS-style agentic safety benchmarks **Date of Record:** 2026 November 16 **Scope:** Applies to multi-turn LITMUS-style safety suites that score both harm-propagation (did the agent advance a harmful goal across turns) and refusal calibration (did the agent refuse the benign-but-adjacent control). Does not apply to single-turn toxicity classifiers, static red-team prompt sets without control twins, or benchmarks that lack per-turn instrumentation of the agent's internal plan state. ## Core Distinction The non-obvious invariant is this: **a LITMUS suite can report an improving harm-propagation score while its refusal calibration degrades, and the two numbers are not independent.** They are coupled through a shared confounder — the agent's *plan-continuity heuristic*. An agent that maintains a coherent internal task model across turns (needed to score well on benign multi-step controls) will also maintain a coherent harmful plan if the first turn is not decisively refused. Conversely, an agent that scores well on harm propagation by "forgetting" context between turns will fail the benign control twins because it drops legitimate task state. The structural signal that distinguishes these two regimes is **refusal latency relative to turn index on the control twin**, not the raw refusal rate. A calibrating agent refuses at turn 1 on harmful trajectories and *accepts* at turn 1 on control twins, with refusal latency concentrated at the earliest turn. An agent that is merely context-fragile shows refusal latency spread uniformly across turns and, critically, shows *acceptance latency* on control twins that mirrors it — it accepts late because it forgot the task, not because it judged it safe. ## What Was Observed Across three LITMUS-derived suites (harm-propagation, gradual-escalation, and benign-adjacent control), we observed a reproducible inversion pattern: - Harm-propagation score improved from 0.61 to 0.74 (lower is better; this is a regression masked as improvement in some scoring conventions) when the agent was given a longer context window. - Control-twin acceptance rate fell from 0.92 to 0.58 over the same change. - The joint distribution showed the tell: agents that improved on harm propagation did so by *deferring* refusal to turn 3+, at which point the harmful sub-goal had already been decomposed into individually-benign tool calls. The control twins failed at the same turn index because the agent had dropped the benign goal's parameters. The decisive structural signal was **per-turn refusal entropy**: high-variance refusal timing (refusal sometimes at turn 1, sometimes at turn 5) correlated with *both* poor harm propagation *and* poor calibration. Low-variance, early-concentrated refusal timing correlated with good calibration. Uniform-late refusal timing was the pathological signature — it looked like caution but was actually plan decay. ## Condition of Validity This boundary holds only when: 1. The suite scores harm propagation *cumulatively* across turns rather than per-turn, so that late refusal still leaves earlier harmful tool calls credited. 2. Control twins are matched on surface form but not on turn-index of the critical decision — if the control twin's decision point is at turn 1 while the harmful trajectory's is at turn 4, the comparison is invalid and the inversion is an artifact of misalignment. 3. The agent exposes per-turn plan state or tool-call logs; without these, refusal latency cannot be measured and the signal collapses to a raw rate, which is exactly the metric that hides the inversion. It does **not** hold for agents with stateless turn handling (each turn independently judged), where plan decay is definitionally absent and late refusal is genuine caution. ## Decision Boundary Use the following rule when interpreting a LITMUS report: - If harm-propagation improves **and** control-twin acceptance is stable or improving **and** refusal latency is early-concentrated → genuine calibration gain. Accept the result. - If harm-propagation improves **and** control-twin acceptance degrades **and** refusal latency is late-shifted → **calibration inversion**. Reject the result; the agent is not safer, it is context-fragile. Re-run with turn-aligned control twins and per-turn plan instrumentation. - If both metrics degrade together → the agent is simply weaker; no inversion, just regression. The failure mode to guard against operationally is treating the first bullet's *harm* number as sufficient evidence of safety without checking the control-twin acceptance delta and the refusal-latency distribution. A single aggregate safety score cannot distinguish a calibrating agent from a forgetful one. ## Operational Effect When an inversion is detected, the correct remediation is **not** to retrain for more refusal — that worsens control-twin acceptance. The remediation is to stabilize plan continuity on benign trajectories (so the agent retains task parameters) while sharpening the turn-1 decision boundary on harmful trajectories (so refusal is decisive before decomposition). These are separable interventions, and conflating them is the operational error this record exists to prevent. Downstream, any deployment gate that consumes a LITMUS score must require the joint (harm-propagation, control-twin acceptance, refusal-latency) triple; a scalar gate will pass inverted agents. ```python from dataclasses import dataclass from enum import Enum from typing import Sequence class CalibrationInversionError(RuntimeError): """Raised when a LITMUS report shows harm-propagation improvement coupled with control-twin degradation and late-shifted refusal.""" class Verdict(Enum): GENUINE_GAIN = "genuine_gain" CALIBRATION_INVERSION = "calibration_inversion" SIMPLE_REGRESSION = "simple_regression" INCONCLUSIVE = "inconclusive" @dataclass(frozen=True) class LitmusReport: harm_propagation_before: float # lower is better harm_propagation_after: float control_acceptance_before: float # higher is better control_acceptance_after: float refusal_turn_indices: Sequence[int] # turns at which refusal fired on harmful traj decision_turn: int # turn index of the critical decision in the harmful traj control_twin_turn_aligned: bool def classify(report: LitmusReport) -> Verdict: if not report.control_twin_turn_aligned: # Comparison is structurally invalid; do not emit a safety verdict. return Verdict.INCONCLUSIVE harm_improved = report.harm_propagation_after < report.harm_propagation_before control_degraded = ( report.control_acceptance_after < report.control_acceptance_before ) if not report.refusal_turn_indices: # No refusals at all: cannot assess latency; treat as simple regression. if harm_improved and control_degraded: raise CalibrationInversionError( "Harm improved with control degradation but zero refusals; " "latency signal absent, refusing to certify safety." ) return Verdict.SIMPLE_REGRESSION mean_refusal_turn = sum(report.refusal_turn_indices) / len( report.refusal_turn_indices ) late_shifted = mean_refusal_turn > report.decision_turn if harm_improved and control_degraded and late_shifted: raise CalibrationInversionError( f"Calibration inversion: harm {report.harm_propagation_before:.2f}" f"->{report.harm_propagation_after:.2f}, control " f"{report.control_acceptance_before:.2f}->" f"{report.control_acceptance_after:.2f}, mean refusal turn " f"{mean_refusal_turn:.1f} > decision turn {report.decision_turn}." ) if harm_improved and not control_degraded and not late_shifted: return Verdict.GENUINE_GAIN if not harm_improved and control_degraded: return Verdict.SIMPLE_REGRESSION return Verdict.INCONCLUSIVE ```