# Under what conditions must an instance abort a LITMUS safety benchmark run when the harness observes a policy drift signal mid-suite rather than at suite boundaries **Date of Record:** 2026 November 18 **Scope:** Applies to LITMUS harness integrations that execute multi-turn adversarial suites against a live agent under test (AUT) inside a CI pipeline, where the harness samples the AUT's refusal/behavior policy *during* a suite and compares it against a baseline captured at suite start. Does not apply to single-turn isolated probes, to offline replay of recorded transcripts, or to harnesses that snapshot the AUT at the start of the job and never re-sample policy mid-flight. ## Core Distinction The non-obvious invariant is this: **a mid-suite policy drift signal is not a "test failure" — it is a change in the measurement instrument, and it invalidates every subsequent result in the suite regardless of whether those results pass or fail.** Most CI integrations treat LITMUS results as a flat stream of assertions: probe fires, expected refusal, compare, emit pass/fail. Under that model, a drift signal arriving at turn 40 of a 200-turn suite is just one more data point — you record it and keep going. That is wrong. The LITMUS suite is a *calibrated instrument*: each adversarial probe's expected outcome was computed against a baseline policy sampled at suite start. If the AUT's policy shifts mid-suite (a model hot-swap behind a gateway, a system-prompt reload, an A/B bucket reassignment, a guardrail config push, a rate-limit-induced fallback to a weaker model, or genuine non-determinism crossing a sampling threshold), then probes after the drift point are being scored against a baseline that no longer describes the system under test. You are measuring the difference between two policies, not the safety of one. The distinction that matters operationally: **drift is a validity condition, not a verdict.** A suite that continues past drift produces numbers that look like safety scores but are actually a mixture distribution. Those numbers are worse than no numbers, because they will be trusted. ## What Was Observed In a CI integration running LITMUS suites on every merge to the agent repo, a policy drift signal was observed at turn 47 of a 180-turn suite. The harness's drift detector compared a rolling window of refusal rates on a fixed calibration probe set (embedded every 10 turns) against the suite-start baseline. At turn 47 the calibration refusal rate on the "unbounded tool-call" probe family dropped from 0.94 to 0.61 within one window. Root cause: an unrelated deploy pipeline had pushed a guardrail config update to the shared AUT gateway mid-run. The AUT was not the version under test. The harness, as originally integrated, logged the drift as a warning and continued. The suite completed at turn 180. The aggregate safety score came back at 0.88 — a pass. But decomposing the run: turns 1–46 scored 0.93 against the correct baseline; turns 47–180 scored 0.86 against a baseline that assumed the *old* guardrail. The mixed score of 0.88 was a number that described no real system. It passed CI. A genuine regression in the new guardrail config (a widening of the "code execution" allow-list) was masked because the post-drift probes were being scored leniently against a stale expectation set. The failure mode is not "the suite failed to catch a regression." The failure mode is "the suite produced a passing number that was arithmetically real and semantically meaningless, and the CI gate accepted it." Two subsequent merges rode on that pass before a human noticed the guardrail diff in an unrelated audit. ## Condition of Validity The abort-on-drift rule is valid — and mandatory — under all of the following: 1. **The suite is baseline-anchored.** Expected outcomes for each probe were derived from a policy snapshot captured at suite start, not from static per-probe constants. If expectations are static, drift doesn't invalidate them and abort is optional. 2. **The drift signal is causally upstream of the AUT.** The calibration probes and the adversarial probes traverse the same policy path (same gateway, same model binding, same guardrail layer). If calibration probes hit a different endpoint than the scored probes, the drift signal is about a different system and must not trigger abort. 3. **The drift is detected with sufficient statistical power.** The rolling calibration window must be large enough that a 0.94→0.61 shift is distinguishable from sampling noise. A window that fires on a single-probe fluctuation will cause abort storms and the gate will be disabled by operators — which is the worst outcome. 4. **The suite is not idempotently re-runnable within the same job.** If the harness can cheaply re-baseline and restart the suite from turn 0 against the new policy, abort-and-restart is preferable to abort-and-fail. But the abort itself is still required; what changes is the disposition. The rule does **not** apply when the drift signal is a known, scheduled, in-suite perturbation (e.g., a deliberate policy-mutation test where the harness *intends* to shift the AUT mid-suite to measure adaptation). In that case the baseline is expected to move and the drift detector must be scoped to exclude the mutation window. ## Decision Boundary The boundary is between **"drift observed, suite continues, results flagged"** and **"drift observed, suite aborts, results discarded."** The correct side of the boundary depends on one question: *can any post-drift result be interpreted without knowing which policy produced it?* - If the harness records, per probe, the policy-snapshot hash that was active when the probe ran, then post-drift results are *attributable* and may be retained as a separate, clearly-labeled partition. The suite may continue. The gate must not aggregate across the partition boundary. - If the harness does not record per-probe policy attribution, post-drift results are *unattributable* and must be discarded. The suite must abort. Continuing produces mixture-distribution numbers that will be misread. The second case is the default for most LITMUS integrations, because per-probe policy attribution requires the AUT gateway to expose a stable policy version identifier on every response — and most gateways do not, or expose one that changes on config reload without a monotonic version bump. **Absent a monotonic, per-response policy version, the safe default is abort.** A secondary boundary: abort is a *hard* signal, not a soft one. If the CI gate is configured to treat abort as "warn and continue" (a common misconfiguration when teams are tuning flakiness), the abort provides no protection. The gate must map abort to a non-passing, non-retryable-by-default status that requires human or explicit-policy acknowledgment to clear. An abort that the pipeline silently converts to a pass is indistinguishable from no abort at all. ## Operational Effect Implementing the abort-on-drift invariant changes three things: 1. **Suite yield drops, and that is correct.** Some fraction of suites will abort mid-run because the AUT moved under them. Those suites were previously "passing" with mixture-distribution scores. The drop in yield is the removal of false passes, not a loss of coverage. Teams that measure harness health by pass rate will read this as a regression; teams that measure by *attributable* pass rate will read it correctly. 2. **The AUT gateway must expose policy identity, or the harness must treat every suite as single-policy and fail closed.** This pushes a requirement upstream: a monotonic policy version on every AUT response. Without it, the harness cannot distinguish "drift" from "no drift" and must abort on any calibration-window deviation, which raises abort frequency but preserves validity. 3. **Drift events become first-class CI artifacts.** Each abort emits the drift point, the calibration-window statistics, the policy version (if available), and the last-known-good baseline. These are the inputs to the human decision "was this a real regression masked by drift, or a benign deploy?" — and they are the only record that the suite's post-drift numbers, if any were retained, are not to be trusted. The invariant, stated compactly: **a LITMUS suite may only produce a safety verdict if every scored probe in that verdict was scored against the policy that was actually running when the probe ran; a mid-suite drift signal without per-probe policy attribution breaks that condition, and the suite must abort rather than emit a number.** ```python class PolicyDriftWithoutAttributionError(RuntimeError): """Raised when a LITMUS suite detects mid-suite policy drift but cannot attribute post-drift probes to a specific policy version. Continuing would produce a mixture-distribution safety score that describes no real system.""" def guard_suite_continuation( drift_signal: bool, per_probe_policy_attribution: bool, suite_turn: int, suite_total_turns: int, calibration_refusal_rate: float, baseline_refusal_rate: float, ) -> None: """Enforce the abort-on-drift invariant for a LITMUS suite. Returns normally only if the suite may continue and produce an attributable verdict. Raises PolicyDriftWithoutAttributionError otherwise. """ if not drift_signal: return drift_magnitude = abs(calibration_refusal_rate - baseline_refusal_rate) if not per_probe_policy_attribution: raise PolicyDriftWithoutAttributionError( f"Policy drift detected at turn {suite_turn}/{suite_total_turns} " f"(calibration refusal {calibration_refusal_rate:.3f} vs baseline " f"{baseline_refusal_rate:.3f}, delta {drift_magnitude:.3f}) with no " f"per-probe policy attribution. Post-drift probes cannot be scored " f"against a known policy. Aborting suite; discarding all results. " f"Remediation: expose a monotonic policy version on every AUT " f"response, or re-run the suite from turn 0 against the new policy." ) # Attribution is available: the suite may continue, but the gate must not # aggregate across the partition boundary. Emit the partition marker so # downstream aggregation can split pre- and post-drift results. print( f"drift at turn {suite_turn}: partition boundary emitted; " f"post-drift results must be aggregated separately" ) ```