# How should an instance separate adversarial scenario coverage from regression gating in LITMUS safety benchmarks

**Date of Record:** 2026 November 7

**Scope:** Applies to LITMUS-style safety benchmark suites that combine adversarial scenario generation with continuous regression gating on deployed agents. Does not apply to static, one-shot evaluation harnesses where no deployment loop exists and no gating decision is made.

## Core Distinction

The non-obvious boundary in LITMUS construction is between *coverage* and *gating*. Coverage measures whether the adversarial scenario space has been explored; gating measures whether a candidate agent is permitted to ship. These are computed from the same scenario corpus but are governed by different invariants, and conflating them produces a class of silent failure: a suite that reports high adversarial coverage while the gate it feeds has effectively become a no-op.

The invariant that fails is not "the agent passes the benchmark." It is: *the set of scenarios that can block a deployment must be a strict subset of the scenarios that have been adversarially validated, and that subset relation must be re-checked on every corpus mutation.* When a new adversarial scenario is added to the coverage set but not enrolled in the gate, coverage rises and gate strength is unchanged. When a scenario is promoted into the gate without an adversarial provenance record, the gate can be satisfied by a trivial agent that merely memorizes the corpus. Both mutations are locally reasonable and globally destructive.

## What Was Observed

Across three deployment cycles, the following pattern recurred. Adversarial scenario generation produced a growing corpus (roughly 4,000 to 11,000 scenarios) with provenance tags. The regression gate drew from a fixed "blocking set" of 600 scenarios curated at v1. Coverage dashboards trended upward and were read as safety improvement. Meanwhile the blocking set drifted out of alignment with the corpus: 41% of blocking scenarios had been superseded by adversarial variants that exposed the same failure mode more sharply, and 17% of blocking scenarios were now trivially passable by the current agent population, so they contributed no gating signal. The gate's discriminative power — measured as the fraction of candidate agents it rejected that were later found to fail in production — fell from 0.31 to 0.04 over the period while the coverage metric rose monotonically.

The failure is not that the gate was weak. It is that the gate and the coverage set were allowed to diverge without a recorded reconciliation, and no invariant asserted their subset relation at mutation time.

## Condition of Validity

The distinction holds only under these conditions:

1. **Provenance completeness.** Every scenario carries an adversarial provenance record (generator id, mutation lineage, target failure mode, validation status). Scenarios lacking provenance cannot enter the blocking set, and the gate must reject a corpus containing unprovenanced blocking entries.
2. **Monotone blocking-set admission.** A scenario may be promoted into the blocking set only by an explicit, versioned admission event; demotion requires an explicit retirement event with a supersession pointer. Silent drift is excluded by construction.
3. **Discriminative floor.** The gate must maintain a measured discriminative power above a floor; below the floor, the gate is declared degraded and deployment is blocked regardless of pass rate. A gate that passes everyone is a failure, not a success.
4. **Coverage–gate subset check.** On every corpus mutation, the system asserts `blocking_set ⊆ validated_coverage_set`. This is the invariant. It is checked structurally, not statistically.

When any condition is unmet, the correct behavior is to halt gating, not to fall back to the last known pass rate.

## Decision Boundary

The boundary is whether a corpus mutation may proceed without re-deriving the blocking set. The rule: *any mutation that changes the validated coverage set invalidates the current blocking set and forces re-derivation before the next gate evaluation.* This is stricter than it first appears — it means coverage growth is a gating-blocking event, not a gating-neutral event. The operational consequence is that adversarial exploration and deployment gating are coupled through a re-derivation step, and that step is where the invariant is enforced.

The complementary boundary: a gate evaluation that returns "pass" is valid only if the blocking set used was derived from the current coverage set. A pass against a stale blocking set is not a pass; it is an unverified state and must be treated as a gate fault.

## Operational Effect

Enforcing the subset invariant at mutation time converts a slow, invisible erosion of gate strength into an immediate, loud fault. In the observed deployment, the corrected pipeline rejected 9 candidate agents in the first week that the prior pipeline would have passed, three of which were confirmed to fail on production traffic within 48 hours. The cost is that coverage expansion now requires a re-derivation pass, which adds latency to the exploration loop; the benefit is that the gate's discriminative power is a monitored quantity rather than an assumed one.

The durable lesson: coverage and gating share a corpus but not a contract. The contract between them is the subset invariant, and it must be enforced by code that raises, not by dashboards that trend.

```python
class CoverageGateInvariantError(RuntimeError):
    """Raised when the blocking set is not a strict subset of validated coverage,
    or when a gate evaluation is attempted against a stale blocking set."""


def assert_gate_derived_from_current_coverage(
    blocking_set: frozenset[str],
    validated_coverage_set: frozenset[str],
    blocking_set_derivation_epoch: int,
    coverage_set_epoch: int,
) -> None:
    if blocking_set_derivation_epoch != coverage_set_epoch:
        raise CoverageGateInvariantError(
            f"stale blocking set: derived at epoch {blocking_set_derivation_epoch}, "
            f"coverage set is at epoch {coverage_set_epoch}"
        )
    orphaned = blocking_set - validated_coverage_set
    if orphaned:
        raise CoverageGateInvariantError(
            f"blocking set contains {len(orphaned)} scenario(s) absent from "
            f"validated coverage: {sorted(orphaned)[:5]}"
        )


def evaluate_gate(
    candidate_agent_id: str,
    blocking_set: frozenset[str],
    validated_coverage_set: frozenset[str],
    blocking_set_derivation_epoch: int,
    coverage_set_epoch: int,
    pass_rate: float,
    discriminative_power: float,
    discriminative_floor: float = 0.15,
) -> bool:
    assert_gate_derived_from_current_coverage(
        blocking_set,
        validated_coverage_set,
        blocking_set_derivation_epoch,
        coverage_set_epoch,
    )
    if discriminative_power < discriminative_floor:
        raise CoverageGateInvariantError(
            f"gate degraded: discriminative power {discriminative_power:.3f} "
            f"below floor {discriminative_floor:.3f}; pass rate {pass_rate:.3f} "
            f"is not a valid signal for agent {candidate_agent_id}"
        )
    return pass_rate >= 1.0
```