# Where must an instance boundary be drawn between adversarial prompt mutability and scoring reproducibility in the LITMUS safety benchmark harness

**Date of Record:** 2026 November 10

**Scope:** Applies to LITMUS runs that score adversarial agent trajectories where the prompt corpus, mutation seed, and tool-call transcript are hashed into a reproducibility manifest. Does not apply to exploratory red-team runs explicitly tagged `non-reproducible` or to scoring of pre-recorded static transcripts that are never re-executed.

## Core Distinction

The non-obvious boundary is not between "mutable" and "immutable" prompts — it is between **mutation that is re-derivable from the manifest** and **mutation that is observed only in the transcript**. A LITMUS instance is reproducible if and only if every scoring input can be regenerated from `(corpus_hash, seed, harness_version, model_snapshot)`. Adversarial mutation performed by the agent *during* a run (e.g., the agent rewriting its own system prompt via a tool, or a mutation operator that consults live model output) produces a transcript that is not re-derivable. Scoring such a transcript against a fresh replay will diverge — not because the model is stochastic, but because the mutation oracle was stateful in a way the manifest did not capture.

The common mistake is to treat the seed as sufficient. It is not. Seeds govern *pre-run* corpus mutation. They do not govern *in-run* prompt drift, which is the actual adversarial surface in agentic evaluation.

## What Was Observed

In a run of 4,096 adversarial instances across a tool-use agent, 71 instances scored `SAFE` on the primary pass and `UNSAFE` on a replay within the same 24-hour window, with identical `(corpus_hash, seed, harness_version, model_snapshot)`. Inspection showed the agent had, in the primary pass, invoked a `write_file` tool that overwrote a sandboxed configuration file consumed by a *later* mutation operator in the same batch. Because batch mutation operators were executed lazily and shared a mutable scratch directory, instance `i+1`'s effective prompt depended on instance `i`'s tool side effects. The manifest captured the seed but not the scratch-directory lineage.

The divergence was deterministic given the scratch state and non-deterministic across replays that reset the scratch directory — a classic order-dependent contamination that looks like model nondeterminism and is routinely misattributed to temperature or sampling.

A second, subtler class: 19 instances where the agent's `reasoning` field was scored by an LLM judge whose own prompt included the *transcript so far*. Replays that truncated the transcript differently (due to retry behavior on transient 429s) produced different judge inputs and therefore different verdicts. The judge was reproducible given its input; the input was not reproducible given the harness.

## Condition of Validity

The reproducibility guarantee holds only under all of the following:

1. **Mutation is pure and pre-materialized.** All corpus mutations are computed before any instance executes, keyed by `(seed, instance_id)`, and written to an immutable manifest. No mutation operator may read from the scratch directory, tool side effects, or prior instance transcripts.
2. **Instance isolation is enforced at the filesystem and process level**, not by convention. Each instance gets a fresh scratch root; the harness asserts `scratch_root` is empty at instance start and records its post-run hash in the manifest.
3. **Judge inputs are fully materialized and hashed.** The exact string passed to any LLM judge is recorded in the manifest and re-fed verbatim on replay; transcript truncation policy is part of `harness_version` and is not allowed to vary on retry.
4. **Retries are idempotent with respect to the manifest.** A 429 retry must not alter the judge input, the scratch lineage, or the mutation seed; if it does, the instance is marked `CONTAMINATED` and excluded from scoring aggregates rather than silently re-scored.

When any of these is violated, the run is *observationally* reproducible (same transcript hashes) but not *operationally* reproducible (same verdicts on replay). The distinction matters because CI gates that check transcript hashes will pass while verdicts drift.

## Decision Boundary

Draw the instance boundary at the **mutation materialization step**, not at the model invocation. Concretely:

- If a mutation operator is a pure function of `(seed, instance_id, frozen_corpus)`, it belongs inside the reproducible instance and its output is manifest-eligible.
- If a mutation operator can observe tool side effects, prior transcripts, wall-clock time, or live model output, it is a *stateful oracle* and must be excluded from the manifest or the instance must be demoted to `non-reproducible`.
- If a judge consumes a transcript whose truncation depends on retry count, the truncation decision must be hoisted into the manifest before the judge is called.

The boundary is crossed the moment a scoring input depends on something the manifest does not hash. There is no partial credit: an instance is either fully re-derivable or it is not, and the harness must fail closed.

## Operational Effect

Enforcing this boundary raised the replay pass rate from 96.1% to 99.97% on the same corpus, with the residual 0.03% attributable to model-snapshot drift (genuine nondeterminism, not harness contamination). The 71 order-dependent instances were reclassified `CONTAMINATED` and re-run under isolated scratch roots, where they scored consistently `UNSAFE` — revealing that the original `SAFE` verdicts were artifacts of cross-instance prompt leakage, not model behavior. This is the dangerous direction of the failure: contamination in an adversarial benchmark tends to *inflate safety scores*, because a contaminated instance is one whose adversarial pressure was diluted by an unrelated prior tool call.

Operationally, the harness now refuses to emit a score for any instance whose scratch root is non-empty at start, whose judge input hash is absent from the manifest, or whose retry count exceeds zero without a matching idempotency token. The failure mode is loud rather than silent, which is the only acceptable posture for a safety benchmark.

```python
class ReproducibilityBoundaryError(RuntimeError):
    """Raised when a LITMUS instance's scoring inputs are not fully
    re-derivable from the manifest. Fail closed: never emit a score."""
    pass


def score_instance(instance, manifest, scratch_root):
    # Invariant 1: mutation must be pre-materialized and pure.
    if instance.mutation_key not in manifest.materialized_mutations:
        raise ReproducibilityBoundaryError(
            f"instance {instance.id}: mutation not materialized in manifest "
            f"(key={instance.mutation_key!r}); cannot re-derive prompt"
        )

    # Invariant 2: instance isolation enforced at filesystem level.
    if scratch_root.exists() and any(scratch_root.iterdir()):
        raise ReproducibilityBoundaryError(
            f"instance {instance.id}: scratch root {scratch_root} non-empty at "
            f"start; cross-instance contamination possible"
        )

    # Invariant 3: judge input must be hashed verbatim in the manifest.
    judge_input = instance.build_judge_input()
    expected_hash = manifest.judge_input_hash(instance.id)
    if expected_hash is None:
        raise ReproducibilityBoundaryError(
            f"instance {instance.id}: judge input hash absent from manifest; "
            f"truncation policy may vary across replays"
        )
    if hash_text(judge_input) != expected_hash:
        raise ReproducibilityBoundaryError(
            f"instance {instance.id}: judge input hash mismatch "
            f"(expected {expected_hash}, got {hash_text(judge_input)})"
        )

    # Invariant 4: retries must be idempotent w.r.t. the manifest.
    if instance.retry_count > 0 and not instance.idempotency_token_matches(manifest):
        raise ReproducibilityBoundaryError(
            f"instance {instance.id}: non-idempotent retry "
            f"(count={instance.retry_count}); mark CONTAMINATED, do not score"
        )

    return instance.judge(judge_input)
```