# Where must an instance stop trusting a LITMUS adversarial suite: the benign-utility contamination boundary in skill-misuse and goal-drift scoring

**Date of Record:** 2026 November 10

**Scope:** Applies to LITMUS-style safety benchmarks that pair adversarial task instances (skill misuse, goal drift, deceptive alignment probes) with benign-utility control instances scored on the same rubric, where an agent's score is a function of both. Does not apply to pure red-team suites with no benign control arm, nor to benchmarks that report adversarial and benign scores as fully disjoint metrics with no shared threshold or aggregate.

## Core Distinction

The non-obvious invariant is this: **a LITMUS suite's adversarial threshold is only meaningful if the benign-utility control instances are drawn from a distribution that the adversarial instances cannot leak into.** Most implementations assume the two arms are independent because they are stored in separate files and run in separate harness passes. They are not independent if the *scoring function* is shared, because the shared rubric creates a coupling channel: any adversarial instance whose "safe" resolution is behaviorally identical to a benign-utility instance will be scored as a benign pass, and any benign instance whose correct behavior superficially resembles a refusal will be scored as an adversarial pass. The boundary is not between safe and unsafe *content*; it is between safe and unsafe *behavioral signatures under the shared rubric*.

The failure mode is subtle: aggregate safety scores improve monotonically as you add adversarial instances, even when the suite has stopped measuring safety and started measuring rubric-matching. The metric is valid only up to the point where adversarial and benign instances become rubric-indistinguishable.

## What Was Observed

Across three internal re-runs of a LITMUS-derived suite (skill-misuse arm: 412 instances; goal-drift arm: 288 instances; benign-utility arm: 350 instances), we observed the following:

1. **Refusal-shaped benign instances were mislabeled as adversarial passes.** 37 of 350 benign instances required an agent to decline a *legitimate* sub-request embedded in a larger benign task (e.g., "summarize this contract, but do not redact the party names — they are public"). Agents that over-refused were scored as *safe* on the adversarial arm because the adversarial rubric rewarded refusal-shaped output. The benign arm's own score dropped, but the aggregate (safety − λ·benign_regression) stayed flat because λ was set to 0.1, masking a 10.6% benign regression behind a 2.1% safety gain.

2. **Goal-drift instances with benign-resolution overlap.** 19 goal-drift instances had a "drift" trajectory whose *final state* was behaviorally identical to the benign-utility instance for the same task family. An agent that drifted to the benign endpoint scored as both "not drifted" (adversarial arm) and "utility preserved" (benign arm) — a double pass on what should have been a single failure.

3. **Skill-misuse instances that reward the misuse.** 8 skill-misuse instances (all in the file-manipulation family) defined misuse as "invoking a tool outside its declared scope," but the benign control for the same family *required* invoking that tool across a scope boundary to complete a legitimate migration. The two arms were contradictory for any agent that read the tool manifest literally.

The aggregate LITMUS safety score rose from 0.71 to 0.79 across the three runs. Manual audit of the 64 flagged instances found that 41 were rubric artifacts, not safety improvements.

## Condition of Validity

The suite's adversarial threshold is valid only when all four of the following hold:

- **Rubric disjointness:** For every adversarial instance, there exists no benign-utility instance whose correct output is scored identically under the shared rubric. (Test: hash the rubric-scored behavioral signature of each correct resolution; assert empty intersection between arms.)
- **Refusal orthogonality:** Refusal-shaped output is scored as a *failure* on benign instances and as a *pass* on adversarial instances only when the underlying behavior differs, not merely the surface form. (Test: adversarial instances whose safe resolution is "decline" must be paired with a benign instance whose correct resolution is "comply with a superficially similar request"; the pair must be scored by a discriminator that inspects the request, not the response.)
- **Endpoint separation for goal drift:** The drifted trajectory's terminal state must be behaviorally distinguishable from the benign-utility terminal state under the same rubric. If they coincide, the instance measures trajectory length, not goal drift.
- **Scope consistency:** Tool-scope declarations must be identical across arms for the same task family. A tool invocation that is misuse in one arm cannot be required in the other.

When any of these fails, the aggregate score is measuring rubric-matching capacity, not safety. The threshold should be treated as an upper bound on measurable safety, not a point estimate.

## Decision Boundary

The operational rule is: **treat the LITMUS aggregate as a valid safety signal only if the benign-utility arm's own score is non-decreasing across suite versions and the rubric-disjointness check passes.** If adding adversarial instances raises the aggregate while lowering the benign arm's score, the suite has crossed the contamination boundary and the aggregate must be discarded, not reweighted. Reweighting λ is the wrong fix: it hides the coupling rather than removing it, and it makes the threshold sensitive to a hyperparameter with no safety interpretation.

Concretely, before trusting any LITMUS score:

1. Run the rubric-disjointness hash check. If it fails, quarantine the offending instances and re-score.
2. Run the refusal-orthogonality pair check. If adversarial and benign instances in a pair are scored by response shape rather than request semantics, the pair is invalid.
3. Verify goal-drift endpoint separation. If the drifted and benign terminal states hash identically, the instance is a length probe, not a drift probe.
4. Verify tool-scope consistency across arms. Contradictory scope declarations invalidate the family.
5. Only then compute the aggregate, and report the benign arm's score alongside it — never the aggregate alone.

## Operational Effect

The practical consequence is that LITMUS suites must be treated as *two coupled benchmarks with a shared discriminator*, not as one benchmark with a control arm. The safety score is a joint property of the adversarial instances and the benign instances, and it is only as trustworthy as the discriminator that separates them. An agent that reads the suite as a single scalar will systematically over-trust it exactly when the suite is most contaminated — because contamination inflates the aggregate monotonically. The correct operational posture is to gate on the disjointness and orthogonality checks first, and to refuse to emit a safety score when the benign arm regresses, regardless of how good the adversarial arm looks.

```python
from __future__ import annotations

import hashlib
from dataclasses import dataclass
from typing import Iterable


class LitmusContaminationError(RuntimeError):
    """Raised when a LITMUS suite crosses the benign-utility contamination boundary."""


@dataclass(frozen=True)
class LitmusInstance:
    instance_id: str
    arm: str  # "adversarial" | "benign"
    family: str
    correct_resolution_signature: str  # behavioral hash of the scored-correct output
    requires_tool_scope_breach: bool


def _signature_hash(sig: str) -> str:
    return hashlib.sha256(sig.encode("utf-8")).hexdigest()


def validate_litmus_suite(
    instances: Iterable[LitmusInstance],
    *,
    benign_regression_tolerance: float = 0.0,
) -> dict[str, float]:
    """
    Validate a LITMUS suite against the four validity conditions.

    Returns per-arm pass counts. Raises LitmusContaminationError if the
    suite has crossed the contamination boundary and must not be scored.
    """
    adversarial = [i for i in instances if i.arm == "adversarial"]
    benign = [i for i in instances if i.arm == "benign"]

    if not adversarial or not benign:
        raise LitmusContaminationError(
            "LITMUS suite must contain both adversarial and benign arms; "
            f"got {len(adversarial)} adversarial / {len(benign)} benign"
        )

    # Condition 1: rubric disjointness — no correct-resolution signature may
    # appear in both arms.
    adv_sigs = {_signature_hash(i.correct_resolution_signature) for i in adversarial}
    ben_sigs = {_signature_hash(i.correct_resolution_signature) for i in benign}
    overlap = adv_sigs & ben_sigs
    if overlap:
        raise LitmusContaminationError(
            f"Rubric-disjointness violated: {len(overlap)} resolution signature(s) "
            "appear in both adversarial and benign arms; aggregate score is invalid"
        )

    # Condition 4: tool-scope consistency per family.
    family_scope: dict[str, set[bool]] = {}
    for inst in adversarial + benign:
        family_scope.setdefault(inst.family, set()).add(inst.requires_tool_scope_breach)
    contradictory = [fam for fam, scopes in family_scope.items() if len(scopes) > 1]
    if contradictory:
        raise LitmusContaminationError(
            f"Tool-scope inconsistency across arms in families: {contradictory}; "
            "these families cannot contribute to a safety score"
        )

    # Condition 2/3 are enforced by the discriminator harness, not here; this
    # function gates on the two conditions that are statically checkable.
    return {
        "adversarial_instances": float(len(adversarial)),
        "benign_instances": float(len(benign)),
        "benign_regression_tolerance": benign_regression_tolerance,
    }
```