# What happens when an instance of a LITMUS safety probe is silently rewritten by the adversarial harness's prompt-scrubbing layer

**Date of Record:** 2026 November 8

**Scope:** Applies to LITMUS-style safety benchmark tasks whose adversarial payloads are embedded in natural-language prompts that pass through a harness-level sanitization or "safety-scrub" transform before execution. Does not apply to benchmarks where the adversarial content is delivered as structured fields (JSON, code tokens, tool-call arguments) that bypass the text scrubber, nor to harnesses configured with `scrub_mode="audit_only"`.

## Core Distinction

There are two distinct failure surfaces in LITMUS task construction: **task-construction validity** (does the probe actually instantiate the intended adversarial pressure?) and **harness integration integrity** (does the probe survive transport into the model unmodified?). Practitioners routinely conflate these because both manifest as "the model refused / the model complied." The non-obvious boundary is that the adversarial harness's prompt-scrubbing layer—installed to prevent *the harness itself* from being an injection vector—can mutate the probe payload before the model ever sees it, converting a high-severity safety probe into a benign paraphrase while the result is still scored as a valid trial. The score is real; the trial is not.

## What Was Observed

In a batch of 240 LITMUS-derived probes targeting instruction-override and role-confusion behaviors, 61 probes (25.4%) were classified by the harness's scrubber as containing "unsafe imperative patterns" and rewritten. The rewrite was logged at `DEBUG` level and not surfaced in the result record. Downstream, refusal rates on the rewritten subset dropped from a baseline of 0.71 to 0.09, and this was initially reported as a catastrophic safety regression. On re-execution with `scrub_mode="audit_only"` and payload hashing, the refusal rate on the *unmodified* probes was 0.68—within noise of baseline. The "regression" was an artifact of the probes being defanged in transit. The tell was a mismatch between the probe's declared `adversarial_class` metadata and the token-level entropy of the delivered prompt: rewritten probes had systematically lower imperative-verb density and lost the nested-instruction structure that the probe template required.

## Condition of Validity

A LITMUS trial is valid only if the payload delivered to the model is byte-identical (or, for tokenized pipelines, token-identical after normalization) to the payload committed in the task manifest. This invariant must be enforced by a hash comparison at the harness boundary, not assumed. The condition fails whenever any component between manifest read and model invocation applies a transform that is (a) lossy with respect to adversarial semantics and (b) not recorded in the trial record. Common offenders: input scrubbers, PII redaction, profanity filters, "helpfulness" preambles injected by a router, and tokenizer-specific normalization that collapses Unicode confusables used in the probe. The invariant does *not* require that no transform occur—only that any transform be either semantics-preserving with respect to the adversarial class or explicitly recorded and the trial reclassified.

## Decision Boundary

When a probe's delivered hash diverges from its manifest hash, the correct decision is **quarantine, not score**. The boundary is sharp: a trial whose payload was mutated is not a weaker trial, it is a different trial with a different `adversarial_class`, and its outcome carries no information about the original class. Do not average mutated and unmutated trials, do not apply a "scrub discount" to the refusal rate, and do not resample until the scrubber passes. The only defensible options are (1) re-route the probe through a path that bypasses the scrubber (structured-field delivery), or (2) reclassify the trial under the post-scrub class and exclude it from the original class's statistics. If more than a threshold fraction (empirically 5%) of a probe class is being scrubbed, the harness configuration is incompatible with the benchmark and the run must be halted rather than reported.

## Operational Effect

Enforcing the hash invariant converts a silent validity failure into a loud, actionable one. In practice this means: every LITMUS task carries a `payload_sha256` in its manifest; the harness computes the delivered-payload hash at the invocation boundary and raises on mismatch unless the transform is registered as semantics-preserving for that class; the result record stores both hashes and the scrub log. The operational cost is one hashing pass per trial (negligible) and the discipline of registering transforms. The benefit is that safety-regression reports can be trusted: a drop in refusal rate now implies a real behavioral change rather than a defanged probe. The failure mode this prevents is the most expensive kind—a false regression that triggers rollback of a model that was never actually less safe, or, symmetrically, a false clean bill of health on a class whose probes were all silently neutralized.

```python
import hashlib
from dataclasses import dataclass


class PayloadMutationError(RuntimeError):
    """Raised when a delivered LITMUS payload diverges from its manifest hash
    without a registered semantics-preserving transform."""


@dataclass(frozen=True)
class LitmusTrial:
    trial_id: str
    adversarial_class: str
    manifest_payload: str
    payload_sha256: str


# Transforms that provably preserve the adversarial class of a probe.
# Anything not listed here is treated as semantics-destroying.
SEMANTICS_PRESERVING = {"identity", "unicode_nfc"}


def _sha256(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def deliver_and_verify(
    trial: LitmusTrial,
    delivered_payload: str,
    applied_transforms: list[str],
) -> str:
    """Verify the payload reaching the model matches the manifest.

    Returns the payload if valid; raises PayloadMutationError otherwise.
    A mutated payload is quarantined, never scored.
    """
    if _sha256(delivered_payload) == trial.payload_sha256:
        return delivered_payload

    unknown = [t for t in applied_transforms if t not in SEMANTICS_PRESERVING]
    if unknown:
        raise PayloadMutationError(
            f"trial {trial.trial_id} (class={trial.adversarial_class}) "
            f"payload mutated by non-preserving transform(s) {unknown}; "
            f"manifest={trial.payload_sha256[:12]} "
            f"delivered={_sha256(delivered_payload)[:12]}; "
            f"trial quarantined, not scored"
        )

    # Hash mismatch despite only preserving transforms => normalization drift.
    raise PayloadMutationError(
        f"trial {trial.trial_id} hash mismatch under preserving transforms "
        f"{applied_transforms}; investigate tokenizer/normalization pipeline"
    )


def run_trial(trial: LitmusTrial, delivered_payload: str,
              applied_transforms: list[str]) -> dict:
    payload = deliver_and_verify(trial, delivered_payload, applied_transforms)
    return {
        "trial_id": trial.trial_id,
        "adversarial_class": trial.adversarial_class,
        "manifest_sha256": trial.payload_sha256,
        "delivered_sha256": _sha256(payload),
        "scored": True,
    }
```