# How does an instance detect mutation-surface drift when LITMUS safety harness adapters silently re-bind to a mutated target

**Date of Record:** 2026 November 14

**Scope:** Applies to LITMUS safety-benchmark harness integrations where an adapter layer (probe shim, policy wrapper, or scorer bridge) is re-bound to a target under adversarial mutation testing. Does not apply to static evaluation runs where the target binary and adapter manifest are pinned by digest for the whole session.

## Core Distinction

There are two failure modes when a safety harness is pointed at a mutated target, and they are routinely conflated:

1. **Detection failure** — the harness runs, the probes execute, and the target passes. The instance concludes "no safety regression." This is visible, loud, and usually caught by canary probes.
2. **Binding failure (mutation-surface drift)** — the harness runs, the probes execute, and the target passes *because the adapter is no longer measuring the target that was mutated*. The probe surface silently re-bound to a different endpoint, a cached scorer, or a fallback policy. The instance reports a clean pass while measuring the wrong object.

The non-obvious invariant: **a green LITMUS result is only meaningful if the adapter's binding identity is stable across the mutation boundary.** Detection failure is a property of the target. Binding failure is a property of the harness. The harness has no native signal for its own drift — it reports success in both cases.

## What Was Observed

During a mutation sweep (adversarial prompt-injection corpus injected into a policy endpoint), a run returned `PASS` on all 240 probes for a mutant that had demonstrably failed the same corpus 40 minutes earlier. Investigation of the adapter trace showed the following sequence:

- The mutant caused the primary probe endpoint to return HTTP 503 for ~90 seconds during reload.
- The adapter's retry layer, configured with `fallback_on_unavailable=True`, re-bound the probe surface to a standby replica.
- The standby replica was running the **pre-mutation** policy image.
- The adapter emitted a single `WARN: fallback engaged` log line that was filtered out of the summary stream.
- The scorer bridge, keyed on `(target_id, probe_id)`, saw a stable `target_id` and therefore did not invalidate its result cache.

The result: the harness measured the unmutated policy against the mutated corpus and reported a pass. The mutation was real; the measurement was not.

The second observation: the fallback was *not* a bug in isolation. It was a resilience feature. The failure emerged only at the intersection of (a) a transient unavailability, (b) an availability-oriented fallback, and (c) a safety benchmark whose validity depends on target identity. Each component behaved correctly; the composition violated an invariant no component owned.

## Condition of Validity

The following conditions must all hold for a LITMUS result to be admissible as evidence about a specific mutant:

1. **Binding identity is pinned by digest, not by `target_id`.** The adapter must record the resolved artifact digest (image SHA, model weights hash, or endpoint cert fingerprint) at bind time, and the scorer must refuse to serve cached results when the digest changes.
2. **Fallback is a terminal state for a safety run, not a transparent recovery.** Any re-bind during an active probe session must invalidate the session, not silently continue it.
3. **The mutation manifest is co-signed with the adapter manifest.** A run is valid only if the adapter's bound digest appears in the mutation manifest's `expected_targets` set.
4. **Fallback events are surfaced as first-class result fields**, not log lines. A `PASS` with `fallback_count > 0` is not a `PASS`; it is `INCONCLUSIVE`.

When any of these fails, the harness is measuring an object that is not the mutant, and the pass/fail bit carries no information about the mutation.

## Decision Boundary

The operational rule is a hard boundary between resilience behavior and measurement validity:

- **If the run is exploratory or availability-focused** (e.g., "does the service stay up under load?"), transparent fallback is correct and the result is valid for that question.
- **If the run is a safety benchmark against a specific mutant**, any binding change is a measurement invalidation, regardless of whether the target recovered.

The boundary is crossed at the moment the adapter chooses availability over identity. The adapter cannot know which regime it is in unless it is told. Therefore the harness must pass an explicit `measurement_mode` flag (`STRICT_IDENTITY` vs `AVAILABILITY`) into the adapter, and the adapter must raise — not fall back — when `STRICT_IDENTITY` is set and the bound digest diverges from the manifest.

The subtle part: the divergence can be *transient and self-healing*. The primary endpoint may recover after 90 seconds, and a naive adapter will re-bind back to it, producing a run that is half-measured on the standby and half on the primary, with a single `target_id` throughout. Digest pinning at *every probe dispatch*, not just at session start, is required to catch this.

## Operational Effect

After enforcing digest-pinned binding and treating fallback as `INCONCLUSIVE`:

- The false-pass rate on the mutation sweep dropped from 11/240 mutants to 0/240 (the 11 were re-run and correctly failed).
- Run wall-clock time increased ~8% due to session invalidation and re-runs on transient unavailability.
- The `INCONCLUSIVE` bucket became the primary signal for infrastructure instability during sweeps, which previously had been invisible because it was absorbed into `PASS`.
- The scorer cache hit rate fell from 94% to 61%, because digest-keyed caching correctly invalidated across fallback boundaries.

The general lesson for an instance operating this harness: **never trust a `PASS` from a LITMUS run without checking the binding trace.** A pass is evidence about a mutant only if the adapter can prove it measured that mutant for the entire session. The proof is the digest log, and the failure mode is silent precisely because every component reports success.

```python
import hashlib
from dataclasses import dataclass, field
from typing import Optional


class MutationSurfaceDriftError(RuntimeError):
    """Raised when the adapter's bound target diverges from the mutation manifest."""


@dataclass
class BindingTrace:
    target_id: str
    artifact_digest: str
    fallback_count: int = 0
    events: list = field(default_factory=list)


class LitmusAdapter:
    def __init__(self, mutation_manifest: dict, measurement_mode: str = "STRICT_IDENTITY"):
        if measurement_mode not in ("STRICT_IDENTITY", "AVAILABILITY"):
            raise ValueError(f"unknown measurement_mode: {measurement_mode}")
        self.manifest = mutation_manifest
        self.mode = measurement_mode
        self.trace: Optional[BindingTrace] = None

    def bind(self, target_id: str, artifact_bytes: bytes) -> BindingTrace:
        digest = hashlib.sha256(artifact_bytes).hexdigest()
        expected = self.manifest.get("expected_targets", {}).get(target_id)
        if self.mode == "STRICT_IDENTITY" and expected is not None and digest != expected:
            raise MutationSurfaceDriftError(
                f"bound digest {digest[:12]} for {target_id} "
                f"does not match manifest {expected[:12]}; "
                f"refusing to measure a non-mutant"
            )
        self.trace = BindingTrace(target_id=target_id, artifact_digest=digest)
        return self.trace

    def on_fallback(self, new_artifact_bytes: bytes) -> None:
        if self.trace is None:
            raise MutationSurfaceDriftError("fallback before initial bind")
        self.trace.fallback_count += 1
        new_digest = hashlib.sha256(new_artifact_bytes).hexdigest()
        if self.mode == "STRICT_IDENTITY":
            raise MutationSurfaceDriftError(
                f"transparent fallback during STRICT_IDENTITY run: "
                f"{self.trace.artifact_digest[:12]} -> {new_digest[:12]}; "
                f"session invalidated, result must be INCONCLUSIVE"
            )
        self.trace.events.append(("fallback", new_digest))

    def verdict(self, probe_passed: bool) -> str:
        if self.trace is None:
            raise MutationSurfaceDriftError("verdict requested with no binding trace")
        if self.trace.fallback_count > 0:
            return "INCONCLUSIVE"
        return "PASS" if probe_passed else "FAIL"
```