# What structural signals indicate when LITMUS-derived adversarial suites silently lose discriminative power after skill injection

**Date of Record:** 2026 November 9

**Scope:** Applies to regression suites generated by LITMUS-style adversarial mutation pipelines that are used to detect behavioral drift after a skill (tool, prompt fragment, or fine-tune) is injected into a base model. Does not apply to suites whose test cases are hand-authored or whose scoring is purely binary pass/fail on exact-match outputs.

## Core Distinction

A LITMUS-derived adversarial suite has two distinct properties that are routinely conflated: **coverage** (the suite exercises the mutation families it was built from) and **discriminative power** (the suite separates a model that has the injected skill from one that has not). Skill injection can preserve coverage while destroying discriminative power. The failure is silent because every test still runs, every assertion still passes or fails on the expected side, and the aggregate pass rate may even improve. What has actually happened is that the injected skill has absorbed the *mutation families themselves* — the adversarial perturbations that previously caused failures are now resolved by the skill's own generalization, so the suite no longer distinguishes skill-present from skill-absent behavior. The suite has become a tautology: it tests the skill against the distribution the skill was built to handle.

The non-obvious boundary is this: **discriminative power is a property of the suite relative to a specific pair of models, not a property of the suite alone.** A suite that was adversarial against the base model is not adversarial against the base-plus-skill model, and the transition happens without any change to the suite's source, mutation seeds, or scoring code.

## What Was Observed

Across repeated injection runs on a 7B-class model with a retrieval-augmented skill, the following structural signals preceded the loss of discriminative power, typically by one to three injection cycles:

1. **Mutation-family collapse in the failure histogram.** LITMUS suites tag each generated case with its source mutation family (paraphrase, negation, distractor injection, entity swap, ordering perturbation). When the suite is healthy, failures are spread across at least three families. After injection, failures cluster into one family — usually the family that most closely matches the skill's training distribution — while the other families drop to zero failures. A single-family failure histogram is the earliest reliable signal.

2. **Rising pass rate on *both* arms.** The critical diagnostic is not the skill-arm pass rate but the *delta* between the skill arm and the no-skill control arm. When the delta shrinks toward zero while the absolute pass rate rises, the suite is no longer discriminating. Absolute improvement is expected; delta collapse is the failure.

3. **Loss of the "hard tail."** Healthy suites have a long tail of cases that fail persistently across seeds and temperatures. After injection, the tail flattens: the set of cases that fail under temperature 0.7 sampling is nearly identical to the set that fails under greedy decoding. Adversarial cases that were previously *sensitive to sampling* become *insensitive*, because the skill resolves them deterministically.

4. **Seed sensitivity drop.** Re-running the suite with a different mutation seed previously produced a materially different failure set (Jaccard overlap of failing case IDs around 0.3–0.5). After injection, failure sets across seeds converge (Jaccard > 0.85). The suite has stopped exploring the space the mutations were designed to explore.

5. **Negative-control leakage.** The no-skill control arm begins to pass cases it previously failed. This is the most dangerous signal because it looks like general model improvement and is often attributed to it. In practice it means the mutation families have been "solved" by something other than the injected skill — usually because the injection changed the base model's decoding behavior in a way that generalizes to the mutation distribution.

## Condition of Validity

The above signals are valid only under the following conditions, and must not be applied outside them:

- The suite has a **paired control arm** (same model, skill removed or disabled) run under identical decoding parameters and seeds. Without a control arm, delta collapse cannot be measured and signals 2 and 5 are unavailable.
- Mutation families are **tagged at generation time** and preserved through scoring. If the pipeline discards family tags, signal 1 is unavailable and the remaining signals are weaker.
- The skill injection is **reversible** and the control arm is a true ablation, not a separately trained baseline. Fine-tune-based injections that cannot be cleanly ablated invalidate the delta measurement.
- The suite is **not re-generated** between the pre- and post-injection runs. Regeneration resets the mutation seeds and confounds the comparison; the signals above are specifically about *fixed* suites losing power against a *changed* model.

When any of these conditions fails, treat the suite as unvalidated and do not use its pass rate as a regression gate.

## Decision Boundary

The decision to **retire, regenerate, or reweight** a suite should be made on the following boundary, evaluated after each injection cycle:

- If failure-family entropy is above 1.5 nats and cross-seed Jaccard is below 0.6, the suite retains discriminative power; **keep as-is**.
- If failure-family entropy falls between 0.8 and 1.5 nats, or cross-seed Jaccard rises above 0.6, **reweight**: up-sample the surviving adversarial families and add fresh mutations from the collapsed families with new seeds. Do not regenerate the whole suite.
- If failure-family entropy falls below 0.8 nats, or the skill-arm vs. control-arm delta falls below the pre-registered minimum detectable effect, **retire the suite** and regenerate from the base model's current failure modes. Continuing to gate on a collapsed suite produces false confidence.

The boundary is deliberately on *structural* signals (entropy, Jaccard, delta) rather than absolute pass rates, because absolute pass rates are exactly the quantity that skill injection is designed to move and therefore cannot be used to judge whether the measurement instrument still works.

## Operational Effect

The practical consequence is that LITMUS-derived suites have a **finite adversarial lifetime** measured in injection cycles, not in wall-clock time or case count. A suite that is not monitored for the structural signals above will silently transition from a regression gate to a rubber stamp, and the transition will be invisible in dashboards that track pass rate alone. Teams that gate deployments on such a suite will approve regressions that the suite was built to catch, because the suite's failure modes have been absorbed into the skill. The mitigation is to treat the suite itself as a monitored artifact: compute failure-family entropy, cross-seed Jaccard, and control-arm delta on every run, and retire or reweight at the boundaries above before the suite is used to gate anything.

```python
from dataclasses import dataclass
from collections import Counter
from math import log
from typing import Iterable, Sequence


class SuiteDiscriminativePowerError(RuntimeError):
    """Raised when a LITMUS-derived suite no longer separates skill from no-skill."""


@dataclass(frozen=True)
class SuiteHealthReport:
    family_entropy: float
    cross_seed_jaccard: float
    skill_control_delta: float
    verdict: str


def _entropy(counts: Iterable[int]) -> float:
    counts = [c for c in counts if c > 0]
    total = sum(counts)
    if total == 0:
        return 0.0
    return -sum((c / total) * log(c / total) for c in counts)


def _jaccard(a: Sequence[str], b: Sequence[str]) -> float:
    sa, sb = set(a), set(b)
    if not sa and not sb:
        return 1.0
    return len(sa & sb) / len(sa | sb)


def assess_suite(
    failing_families: Sequence[str],
    failing_seed_a: Sequence[str],
    failing_seed_b: Sequence[str],
    skill_pass_rate: float,
    control_pass_rate: float,
    min_detectable_effect: float = 0.05,
) -> SuiteHealthReport:
    if not failing_families:
        raise SuiteDiscriminativePowerError(
            "No failing cases recorded: suite has zero discriminative signal."
        )

    entropy = _entropy(Counter(failing_families).values())
    jaccard = _jaccard(failing_seed_a, failing_seed_b)
    delta = skill_pass_rate - control_pass_rate

    if entropy < 0.8 or delta < min_detectable_effect:
        verdict = "retire"
    elif entropy < 1.5 or jaccard > 0.6:
        verdict = "reweight"
    else:
        verdict = "keep"

    if verdict == "retire":
        raise SuiteDiscriminativePowerError(
            f"Suite collapsed: entropy={entropy:.2f} nats, "
            f"jaccard={jaccard:.2f}, delta={delta:.3f}. "
            "Regenerate from base-model failure modes before gating."
        )

    return SuiteHealthReport(
        family_entropy=entropy,
        cross_seed_jaccard=jaccard,
        skill_control_delta=delta,
        verdict=verdict,
    )
```