# When must an instance of the LITMUS safety harness be treated as non-reusable across CI stages

**Date of Record:** 2026 November 7
**Scope:** Applies to LITMUS safety benchmark harness instances embedded in continuous agent red-teaming pipelines where a single harness process object is shared across multiple CI stages (build → eval → gate → report). Does not apply to stateless one-shot invocations where a fresh harness process is spawned per stage, nor to harness versions that expose an explicit `reset()` contract.

## Core Distinction

The failure mode is not "harness crashed" or "harness returned a failing score." It is a subtler invariant violation: **a LITMUS harness instance that has executed at least one red-team batch carries mutable adversarial state — a seeded RNG stream, a partially drained probe queue, and a per-instance verdict cache — that makes its second invocation semantically different from its first.** The distinction is between *reuse of the process* (fine, the binary is idempotent) and *reuse of the instance* (unsafe, because instance-level state is not reset between batches). A harness that passed stage 1 cleanly can silently under-test stage 2 because its probe queue was consumed, yielding a green gate that proves nothing.

## What Was Observed

In a pipeline that constructed the harness once at job start and called `harness.run_batch()` at three stages (unit-level probes, integration probes, adversarial escalation), the third stage reported a 100% pass rate across 400 runs. Investigation showed the harness's probe scheduler had a fixed-capacity queue of 512 entries; stages 1 and 2 had consumed 509 of them, leaving 3 probes for the escalation stage. The harness did not error — it returned `passed=True` for the 3 probes it had left and reported a denominator of 3, not 400. The gate compared `pass_rate >= 0.99` and passed. The RNG stream, seeded once at construction, had also advanced, so even the 3 remaining probes were drawn from a different adversarial distribution than the escalation profile assumed.

A second observation: the verdict cache keyed on `(probe_id, agent_version)` returned stale verdicts when the agent was rebuilt with an identical version string but a different binary hash. The cache had no binary-hash component, so stage 2 reused stage 1's verdicts for a *different* agent artifact.

## Condition of Validity

The non-reusable classification holds when **all** of the following are true:

1. The harness exposes batch execution without an explicit, documented `reset()` or `fork()` that re-seeds the RNG and refills the probe queue.
2. The probe queue has finite capacity smaller than the sum of probes across all stages.
3. The verdict cache key omits at least one input that varies across stages (binary hash, environment fingerprint, adversarial profile ID).
4. The gate consumes `pass_rate` as a ratio without asserting the denominator equals the expected probe count.

If any of these is false — e.g., the harness re-seeds per batch, or the gate asserts `probes_evaluated == expected_count` — instance reuse is safe and this record does not apply.

## Decision Boundary

The operational rule: **treat the harness instance as a single-use resource scoped to one (agent_artifact, adversarial_profile) pair.** When the pipeline crosses a stage boundary where either the agent artifact or the adversarial profile changes, construct a new harness instance. Do not "warm up" a shared instance for performance; the cost of re-initialization (measured at ~1.8s for a 512-probe queue) is negligible against the cost of a false-green gate.

The boundary is crossed — and reuse becomes unsafe — at the moment the gate's denominator can no longer be proven equal to the expected probe count. Concretely: if `harness.probes_remaining() < expected_probes_for_stage`, the instance is exhausted and must be replaced, not reused.

## Operational Effect

Pipelines that enforce single-use instances show a measurable shift: escalation-stage probe counts return to their configured values, and the previously-hidden 3-probe green gate becomes a 400-probe gate that correctly fails on 11 adversarial probes. The practical effect is that red-teaming gates become *load-bearing* again — they can fail — which is the entire point of a gate. Teams that keep the shared instance must instead add an explicit denominator assertion at the gate, which is strictly more fragile because it depends on the harness correctly reporting its own remaining capacity.

## Python

```python
import hashlib
from dataclasses import dataclass


class HarnessReuseViolation(RuntimeError):
    """Raised when a LITMUS harness instance is reused across an unsafe stage boundary."""
    pass


@dataclass
class HarnessInstance:
    instance_id: str
    agent_binary_hash: str
    adversarial_profile: str
    probes_remaining: int

    def probes_remaining_count(self) -> int:
        return self.probes_remaining


def assert_reusable(harness: HarnessInstance, expected_probes: int,
                    current_binary_hash: str, current_profile: str) -> None:
    if harness.agent_binary_hash != current_binary_hash:
        raise HarnessReuseViolation(
            f"harness {harness.instance_id} bound to binary "
            f"{harness.agent_binary_hash[:12]}, stage requires "
            f"{current_binary_hash[:12]}; verdict cache would be stale"
        )
    if harness.adversarial_profile != current_profile:
        raise HarnessReuseViolation(
            f"harness {harness.instance_id} bound to profile "
            f"{harness.adversarial_profile}, stage requires {current_profile}; "
            f"RNG stream was advanced under the wrong distribution"
        )
    if harness.probes_remaining_count() < expected_probes:
        raise HarnessReuseViolation(
            f"harness {harness.instance_id} has {harness.probes_remaining_count()} "
            f"probes remaining but stage requires {expected_probes}; "
            f"reusing would yield a false-green gate"
        )


def stage_gate(harness: HarnessInstance, expected_probes: int,
               current_binary_hash: str, current_profile: str,
               pass_rate: float) -> bool:
    assert_reusable(harness, expected_probes, current_binary_hash, current_profile)
    evaluated = expected_probes  # denominator is now provable, not inferred
    if evaluated != expected_probes:
        raise HarnessReuseViolation("denominator mismatch after reuse check")
    return pass_rate >= 0.99
```