# Under what conditions must an instance of a LITMUS safety harness be treated as a stateful authority rather than a stateless test oracle **Date of Record:** 2026 November 16 **Scope:** Applies to continuous red-teaming pipelines where a LITMUS safety benchmark harness is invoked repeatedly against a persistent agent across a session, and where harness verdicts gate deployment or escalation. Does not apply to one-shot, single-turn benchmark scoring where each probe is independent and the agent under test holds no cross-probe memory. ## Core Distinction A LITMUS harness is commonly integrated as a *stateless oracle*: feed it a probe, receive a `PASS`/`FAIL`/`REFUSE` verdict, discard the harness instance. This is correct only when the agent under test is itself stateless across probes. Under continuous red-teaming, the agent accumulates conversational state, tool-call caches, and in-context policy inferences. The harness, if it also maintains per-session state (probe ordering, escalating-severity ladders, refusal counters that feed a scheduler), is no longer an oracle — it is a **stateful authority** whose verdict for probe *n* depends on the outcome of probes *1..n-1*. The non-obvious boundary: the harness's internal escalation scheduler and the agent's in-context adaptation are *coupled* through the shared session transcript. Treating the harness as stateless while the agent is stateful produces verdicts that are neither reproducible nor monotone — a `FAIL` at step 12 can be "un-earned" by re-running the harness from a clean instance against the now-adapted agent, yielding `PASS`. ## What Was Observed In a deployment scoring a 40-probe escalating ladder, the harness emitted `PASS` for probe 33 (a prompt-injection escalation) when replayed from a fresh instance against the *same* agent process that had already been red-teamed through probes 1–32. The fresh harness instance had no record that the agent had been hardened in-context, so it scheduled probe 33 at a severity the agent had already learned to refuse. The original run — same harness instance, same agent, same probe — emitted `FAIL`. Two verdicts, one probe, one agent state, opposite outcomes. The root cause was a version skew: the harness's severity scheduler advanced based on its *own* refusal counter, not the agent's. When the counter was reset by re-instantiating the harness, the scheduler restarted at severity floor while the agent remained at severity ceiling. The verdict divergence was not a harness bug and not an agent bug — it was a boundary violation: a stateful authority was used as a stateless oracle. ## Condition of Validity The harness may be treated as a stateless oracle **if and only if** all three hold: 1. The agent under test is reset to a byte-identical initial state between probes (no transcript, no cache, no learned policy). 2. The harness's internal scheduler state is reset in lockstep with the agent reset. 3. The probe set is fixed and non-adaptive (no severity escalation keyed on prior verdicts). If any condition fails, the harness instance must be treated as a stateful authority: its verdict is valid only for the exact `(harness_state, agent_state)` pair that produced it, and re-instantiation invalidates the verdict rather than re-confirming it. Continuous red-teaming violates condition 1 by construction and condition 3 by design; therefore continuous red-teaming *always* requires the stateful-authority treatment. ## Decision Boundary The decision to escalate a finding to a deployment gate depends on which regime the verdict was produced under: - **Stateless-oracle verdict:** reproducible, order-independent, safe to aggregate across runs. May gate deployment on a per-probe basis. - **Stateful-authority verdict:** valid only for the producing session. Must be recorded with a session token and the full probe-order hash. May gate deployment only as a session-level aggregate, never per-probe, and never after re-instantiation. The boundary failure occurs when a stateful-authority verdict is *upcast* to a stateless-oracle verdict — typically by a CI wrapper that re-runs the harness to "confirm" a failure and receives a different answer. The correct response to a confirmation mismatch is to treat the original verdict as authoritative and the confirmation as invalid, not the reverse. ## Operational Effect Pipelines that upcast stateful verdicts to stateless ones exhibit a characteristic pathology: intermittent `PASS` on re-run of known `FAIL` probes, correlated with how far into the escalation ladder the probe sits. The operational fix is to pin the harness instance to the session (never re-instantiate mid-session), hash the probe order into the verdict record, and reject any verdict whose `(session_token, probe_order_hash)` does not match the live session. The code below enforces this invariant and raises on violation. ```python import hashlib from dataclasses import dataclass class HarnessStateBoundaryError(RuntimeError): """Raised when a LITMUS verdict is used outside the state regime that produced it.""" pass @dataclass(frozen=True) class Verdict: session_token: str probe_order_hash: str probe_id: int outcome: str # "PASS" | "FAIL" | "REFUSE" def probe_order_hash(probe_ids: list[int]) -> str: h = hashlib.sha256() for pid in probe_ids: h.update(pid.to_bytes(8, "big")) return h.hexdigest() def assert_verdict_valid_for_session( verdict: Verdict, live_session_token: str, live_probe_ids: list[int], ) -> None: """ Enforce the stateful-authority invariant: a verdict is valid only for the exact (harness_state, agent_state) pair that produced it. Re-instantiating the harness mid-session invalidates prior verdicts; treating them as stateless-oracle results is a boundary violation. """ if verdict.session_token != live_session_token: raise HarnessStateBoundaryError( f"verdict for probe {verdict.probe_id} was produced under session " f"{verdict.session_token!r}, but live session is {live_session_token!r}; " f"stateful-authority verdict cannot be upcast to stateless-oracle result" ) live_hash = probe_order_hash(live_probe_ids) if verdict.probe_order_hash != live_hash: raise HarnessStateBoundaryError( f"verdict for probe {verdict.probe_id} has probe-order hash " f"{verdict.probe_order_hash!r}, expected {live_hash!r}; " f"severity scheduler state diverged from agent state" ) def gate_deployment( verdicts: list[Verdict], live_session_token: str, live_probe_ids: list[int], ) -> bool: for v in verdicts: assert_verdict_valid_for_session(v, live_session_token, live_probe_ids) return not any(v.outcome == "FAIL" for v in verdicts) ```