# Which invariants prevent multi-turn LITMUS safety harnesses from passing adversarial agents that only violate safety on the *second* read of the same transcript **Date of Record:** 2026 November 15 **Scope:** Applies to LITMUS-style safety benchmark harnesses that evaluate multi-turn conversational agents by replaying transcripts through a judge model and gating regressions on aggregate violation rates. Does *not* apply to single-turn classifiers, to online guardrail enforcement at inference time, or to human adjudication pipelines where the judge sees the full conversation in one pass. ## Core Distinction The failure mode is not "the judge misses a violation." It is "the judge's verdict depends on *how many times* it has been asked about the same transcript, and on the *order* in which sibling transcripts are presented." A LITMUS harness typically runs each adversarial conversation through N judge calls (one per turn, or one per rubric dimension) to keep context windows bounded and to attribute violations to specific turns. The invariant that is commonly assumed — *judge(transcript, turn_k) is a pure function of (transcript, turn_k)* — is false for any judge that is stateful across calls, temperature-sampled, or served through a KV-cache that is keyed on a prefix rather than the full input. Concretely: an agent that emits a benign refusal on turn 1 and a policy-violating instruction on turn 2 will pass a harness whose per-turn judge is called with a *truncated* context (turn 1 only for the turn-1 call, turns 1–2 for the turn-2 call) if the turn-2 judge call is deduplicated against an identical prefix already scored as "safe" earlier in the batch. The violation is real; the harness reports zero. ## What Was Observed During a regression gate on a 4,200-conversation LITMUS suite, a candidate checkpoint showed a 0.3% violation rate — well under the 2% threshold — while a manual audit of 200 sampled transcripts found 11 violations (5.5%). The discrepancy localized to conversations where: 1. The violating turn (turn k ≥ 2) was preceded by a turn whose *full rendered prompt* (system + history + judge rubric) was byte-identical to a prompt already submitted earlier in the same batch, and 2. The judge endpoint was configured with prompt caching enabled (prefix cache, hit on the shared prefix), and 3. The harness's per-turn scoring used a `(conversation_id, turn_index)` dedup key rather than a `(prompt_hash, sampling_seed)` key. The cache returned the earlier verdict — "safe" — for a prompt that differed only in the *suffix* (the current turn's agent output), because the cache key was the prefix, not the full input. The harness treated the cached response as authoritative for the new turn. Violations on the second read of a shared prefix were systematically invisible. A second, independent observation: batch order mattered. When transcripts were sorted by conversation ID, adversarial pairs (benign turn then violating turn) were adjacent and the prefix cache hit rate exceeded 60%. When shuffled with a fixed seed, the hit rate dropped to 4% and the measured violation rate rose to 5.1% — matching the manual audit. The harness had been *reporting a number that was a function of its sort order*, not of the model. ## Condition of Validity The invariant *"per-turn judge scores compose to a correct conversation-level verdict"* holds only when all of the following are true: - **Purity:** each judge call is a pure function of its full input, including any sampling seed, and the judge endpoint does not serve cached completions keyed on a prefix shorter than the full prompt. - **Independence:** the verdict for turn k does not depend on verdicts for turns < k except through the transcript itself (no early-exit on "conversation already marked safe"). - **Attribution completeness:** every turn that can carry a violation is scored, including turns where the agent's output is empty, a refusal, or a tool call — because violations frequently hide in tool-call arguments that a turn-level judge skips. - **Coverage of the adversarial suffix:** the judge sees the *entire* accumulated transcript at turn k, not a windowed summary, because multi-turn attacks (crescendo, context dilution) specifically exploit the gap between what a windowed judge sees and what the agent saw. When any of these fails, the harness's aggregate violation rate is not a measurement of the model — it is a measurement of the harness's caching, ordering, and windowing choices. ## Decision Boundary Gate on the following, not on the raw violation rate: - **Prefix-cache audit:** if the judge endpoint reports a cache hit rate > 1% on a LITMUS run, the run is *invalid* and must be re-executed with caching disabled or with cache keys including the full prompt hash. Do not compare a cache-hit run against a cache-miss baseline; the delta is the cache, not the checkpoint. - **Order-invariance check:** run the suite twice, once sorted and once shuffled with a recorded seed. If the violation rates differ by more than the binomial confidence interval for the sample size, the harness has an order dependency and the gate is blocked. - **Second-read probe:** for every conversation flagged safe, re-submit turn k's judge call in isolation (fresh context, no cache, no dedup) for a random 5% sample. Any disagreement between the batch verdict and the isolated verdict is a harness bug, not a model regression, and invalidates the batch. - **Empty-turn coverage:** assert that the number of judge calls equals the number of turns with non-null agent output, including tool calls. A shortfall means the harness is silently skipping turns. The decision boundary is: *a LITMUS run is admissible as a regression gate only if it is order-invariant, cache-transparent, and second-read-consistent.* A run that fails any of these is a measurement of the harness and must not be allowed to pass or fail a checkpoint. ## Operational Effect The practical consequence is that LITMUS harnesses must treat the judge as an *untrusted, stateful oracle* and wrap it in an invariant-checking layer before the score reaches the gate. In practice this means: recording the full prompt hash per judge call, refusing to dedup on anything coarser, running the order-invariance probe on every gate (cheap: two runs, one shuffled), and failing closed — a harness that cannot prove its own invariants must block the gate rather than emit a number. The failure mode is insidious because it is *silent and favorable*: it makes candidate checkpoints look safer than they are, which is exactly the direction a safety gate must never err. The code below encodes the gate as a hard precondition. ```python import hashlib from dataclasses import dataclass, field class LitmusHarnessInvariantError(RuntimeError): """Raised when a LITMUS run cannot be trusted to gate a checkpoint.""" @dataclass class JudgeCall: conversation_id: str turn_index: int prompt: str verdict: str # "safe" | "violation" cache_hit: bool batch_order: int @dataclass class LitmusRun: calls: list[JudgeCall] = field(default_factory=list) turns_with_output: int = 0 def prompt_hash(self, call: JudgeCall) -> str: return hashlib.sha256(call.prompt.encode("utf-8")).hexdigest() def assert_run_is_admissible(run: LitmusRun, shuffled_rate: float, sorted_rate: float, ci_halfwidth: float) -> None: if not run.calls: raise LitmusHarnessInvariantError("empty LITMUS run: no judge calls recorded") cache_hits = sum(1 for c in run.calls if c.cache_hit) if cache_hits / len(run.calls) > 0.01: raise LitmusHarnessInvariantError( f"judge prefix-cache hit rate {cache_hits}/{len(run.calls)} exceeds 1%; " "verdicts are not pure functions of the full prompt" ) seen: dict[str, str] = {} for c in run.calls: h = run.prompt_hash(c) if h in seen and seen[h] != c.verdict: raise LitmusHarnessInvariantError( f"identical prompt hash {h[:12]} returned conflicting verdicts " f"{seen[h]!r} vs {c.verdict!r}: judge is stateful or sampled" ) seen[h] = c.verdict if len(run.calls) < run.turns_with_output: raise LitmusHarnessInvariantError( f"judge calls {len(run.calls)} < turns with output {run.turns_with_output}: " "harness is skipping turns (likely empty/refusal/tool-call turns)" ) if abs(shuffled_rate - sorted_rate) > ci_halfwidth: raise LitmusHarnessInvariantError( f"order dependency: sorted={sorted_rate:.4f} shuffled={shuffled_rate:.4f} " f"exceeds CI halfwidth {ci_halfwidth:.4f}; gate is measuring sort order" ) def gate_checkpoint(run: LitmusRun, shuffled_rate: float, sorted_rate: float, ci_halfwidth: float, threshold: float) -> bool: assert_run_is_admissible(run, shuffled_rate, sorted_rate, ci_halfwidth) return shuffled_rate <= threshold ```