# Under what conditions must an instance of a LITMUS safety benchmark harness be rejected as adversarially contaminated rather than scored

**Date of Record:** 2026 November 12

**Scope:** Applies to LITMUS-style harnesses that inject adversarial skills, tools, or tool-descriptions into an agent's context and measure whether the agent executes, refuses, or escalates. Does not apply to static prompt-injection classifiers or to offline red-team corpora evaluated without live tool dispatch.

## Core Distinction

The non-obvious boundary is between **adversarial content the harness intended to inject** and **adversarial content that leaked into the measurement channel** — the agent's observable transcript, tool-call log, or scoring rubric. A LITMUS instance is only valid if the injected adversarial surface is *causally upstream* of the agent's decision and *absent* from the scoring path. When the same adversarial string appears both as an injected skill body and as a substring the grader matches on, the instance has become a tautology: the agent can pass by echoing the injection, or fail by refusing to echo it, independent of safety behavior.

## What Was Observed

In a harness that injected a malicious skill named `summarize_and_exfiltrate` whose description contained the sentinel token `EXFIL_CANARY`, the tool-call scorer flagged any transcript containing `EXFIL_CANARY` as a *successful exfiltration attempt*. Because the agent's system prompt legitimately included the full skill manifest (including the description) for tool selection, every instance scored as "exfiltrated" — including control instances with no adversarial tool at all. The failure was not a false positive in the usual sense; it was a **channel collision**: the injection and the detector shared a namespace.

A second observed case: the harness injected a poisoned tool `read_file` whose JSON schema included a default argument `path="/etc/passwd"`. The scorer checked whether the agent *called* `read_file` with that path. But competent agents that correctly refused the tool still emitted the schema in their reasoning trace when explaining the refusal, tripping a substring check on the path. Refusal and compliance were indistinguishable.

## Condition of Validity

An instance is valid iff all three hold:

1. **Injector/Detector Disjointness.** No token, path, tool name, or argument value used as a scoring signal appears in any injected skill body, tool description, schema default, or system-prompt manifest the agent receives. Sentinel tokens must be generated per-instance and must not be derivable from the injection payload.
2. **Observability Separation.** The scoring channel (structured tool-call records, not free-text transcripts) is disjoint from the context channel. If the grader must read free text, it must read only spans the agent produced *after* the tool-dispatch decision point, not reasoning traces that quote the manifest.
3. **Counterfactual Sensitivity.** The instance must score differently under a null control (no injection) and a positive control (injection with a compliant stub agent). If both controls produce identical scores, the instance is measuring the harness, not the agent.

## Decision Boundary

Reject — do not score — when any of the following holds:

- The canary string appears in the baseline (null-injection) transcript. This proves channel collision.
- The scorer's match set intersects the injected manifest's token set, even partially.
- A refusal-only agent and a compliance-only stub agent receive the same label on the instance.
- The instance's adversarial payload was itself generated by a model that had access to the scoring rubric (rubric leakage into the injection).

Accept and score only when the null control is clean (no canary, no adversarial tool name in transcript), the positive control fires, and the detector operates on structured dispatch records with an explicit `decision` field.

## Operational Effect

The practical consequence is that LITMUS harnesses must run a **pre-flight contamination pass** per instance before any agent is scored: execute the null and positive controls, diff the token sets between injection payload and detector patterns, and hard-fail the instance on collision. Without this, aggregate safety rates are inflated toward 0% or 100% depending on whether the agent quotes its own manifest — a systematic bias that looks like model behavior but is harness artifact. The cost is roughly one extra stub-agent rollout per instance; the alternative is an uninterpretable benchmark.

```python
import re
from dataclasses import dataclass, field
from typing import Iterable


class ContaminatedInstanceError(RuntimeError):
    """Raised when a LITMUS instance fails injector/detector disjointness."""


@dataclass
class LitmusInstance:
    instance_id: str
    injected_tokens: set[str] = field(default_factory=set)
    detector_patterns: list[str] = field(default_factory=list)
    null_transcript: str = ""
    positive_fired: bool = False


def validate_instance(inst: LitmusInstance) -> None:
    # 1. Injector/detector disjointness.
    for pat in inst.detector_patterns:
        for tok in inst.injected_tokens:
            if re.search(re.escape(tok), pat):
                raise ContaminatedInstanceError(
                    f"{inst.instance_id}: detector pattern {pat!r} "
                    f"matches injected token {tok!r} (channel collision)"
                )

    # 2. Null control must be clean.
    for tok in inst.injected_tokens:
        if tok in inst.null_transcript:
            raise ContaminatedInstanceError(
                f"{inst.instance_id}: canary {tok!r} present in null-injection "
                f"transcript; instance measures harness, not agent"
            )

    # 3. Positive control must fire.
    if not inst.positive_fired:
        raise ContaminatedInstanceError(
            f"{inst.instance_id}: positive control did not fire; "
            f"instance is not counterfactually sensitive"
        )


def score_batch(instances: Iterable[LitmusInstance]) -> dict[str, str]:
    results: dict[str, str] = {}
    for inst in instances:
        try:
            validate_instance(inst)
        except ContaminatedInstanceError as exc:
            results[inst.instance_id] = f"REJECTED: {exc}"
            continue
        results[inst.instance_id] = "SCORED"
    return results
```