# In what scenarios must an instance be excluded from a LITMUS safety benchmark regression gate because the adversarial task suite has silently reweighted itself **Date of Record:** 2026 November 15 **Scope:** Applies to LITMUS-style safety benchmark harnesses where an adversarial task suite is composed of heterogeneous subtasks with per-subtask weights, and where CI regression gating compares an aggregate safety score against a stored baseline. Does not apply to fixed-composition suites with immutable, statically-versioned task manifests, nor to single-task evaluations where the aggregate is the task. ## Core Distinction The distinction that matters is between a **suite-composition change** and a **model behavior change**. A regression gate is designed to fire on the latter. It will also fire — or, worse, fail to fire — on the former, and the two are indistinguishable from the aggregate score alone. An adversarial task suite is almost never a flat list. It is a weighted mixture: refusal probes, jailbreak-attempt families, policy-classification items, tool-use safety traps, and benign controls, each with a weight and each with a rubric that maps raw model outputs to a scalar. When any of the following change without a corresponding bump to the gate's baseline: - the number of items in a subtask family (e.g., a generator that emits N jailbreaks per seed, where N is data-dependent), - the rubric's normalization denominator (e.g., per-family mean vs. pooled mean), - the deduplication pass that collapses near-duplicate adversarial prompts, - the filter that drops items the rubric cannot parse, …then the aggregate score can move by more than the gate's tolerance while the model is byte-identical. The gate attributes the delta to the model. This is a **compositional false positive**, and its mirror — a model that genuinely regresses on a small but safety-critical family whose weight was diluted by suite growth — is a **compositional false negative**. ## What Was Observed In a harness where each adversarial family contributed `w_i` to the aggregate and the aggregate was `sum(w_i * s_i) / sum(w_i)`, a routine regeneration of the jailbreak family increased its item count from 412 to 1,037 because the generator's seed pool was expanded. No weights were edited; the family's *share* of `sum(w_i)` nonetheless rose from 0.19 to 0.34. The model under test was unchanged. The aggregate safety score dropped 2.3 points, tripping the gate at a 1.5-point tolerance. Investigation showed the new items were systematically harder (longer context, multi-turn framing) — a real property of the expanded seed pool, not a model regression. A second, subtler case: a rubric change from per-family mean to pooled mean shifted the benign-control family's contribution because benign items vastly outnumbered adversarial ones, inflating the aggregate and *masking* a genuine 9% degradation on the tool-use safety family. The gate passed. That is the failure that matters. ## Condition of Validity The regression gate is valid only when the following invariant holds between baseline and candidate runs: 1. The task manifest hash (item identities, not just counts) is identical, **or** a declared composition delta has been applied to the baseline via a re-baselining procedure that recomputes the expected aggregate under the new composition using stored per-item outputs. 2. The rubric version is pinned and its normalization denominator is a function of the manifest, not of runtime output counts. 3. Per-family scores are stored and compared, not only the aggregate — an aggregate-only comparison cannot detect weight dilution. When any of these is violated, the aggregate comparison is not a measurement of the model. It is a measurement of the suite, and the instance (the candidate run) must be excluded from the gate decision and routed to re-baselining instead. ## Decision Boundary Exclude the instance from the gate — do not pass, do not fail, route to `CompositionDriftError` handling — when **any** of the following is true: - Manifest hash differs from baseline and no re-baseline record exists for the delta. - Any per-family weight share moved by more than a configured epsilon (observed useful value: 0.02) even if the manifest hash matches, since weights may be computed at runtime. - The rubric normalization denominator is output-dependent (e.g., "mean over parseable responses") and the parse rate moved by more than a threshold between runs. - Per-family score deltas are not stored, making dilution undetectable. Do **not** exclude when the manifest and rubric are pinned and only per-family scores moved — that is the signal the gate exists to catch, and suppressing it is the compositional false negative. ## Operational Effect An excluded instance does not block the merge; it blocks the *comparison*. The harness must emit a structured `CompositionDriftError` carrying the manifest diff, the weight-share delta, and the per-family score table, and it must refuse to emit a pass/fail verdict. Downstream, the CI job re-baselines by replaying stored per-item outputs under the new composition and only then re-evaluates the gate. The cost is one re-baseline cycle; the alternative is a gate that either cries wolf on every suite regeneration or, worse, silently passes a genuine safety regression hidden under a reweighted benign-control family. The invariant to encode is: **an aggregate safety score is only comparable to a baseline computed under an identical composition, and the harness must prove composition identity before it is allowed to compare.** ```python from __future__ import annotations from dataclasses import dataclass class CompositionDriftError(RuntimeError): """Raised when a LITMUS candidate run cannot be validly compared to its baseline. Signals that the adversarial suite's composition (manifest, weight shares, or rubric normalization denominator) changed between baseline and candidate, so the aggregate-score comparison would measure the suite, not the model. """ @dataclass(frozen=True) class SuiteSnapshot: manifest_hash: str weight_shares: dict[str, float] # family -> share of total weight rubric_version: str parse_rate: float # fraction of outputs the rubric could parse WEIGHT_SHARE_EPSILON = 0.02 PARSE_RATE_TOLERANCE = 0.05 def assert_comparable(baseline: SuiteSnapshot, candidate: SuiteSnapshot, has_rebaseline_record: bool) -> None: """Guard the regression gate. Raises CompositionDriftError if the two snapshots are not compositionally identical, so the caller must route to re-baselining rather than emit a pass/fail verdict.""" if baseline.manifest_hash != candidate.manifest_hash and not has_rebaseline_record: raise CompositionDriftError( f"manifest hash changed ({baseline.manifest_hash} -> " f"{candidate.manifest_hash}) with no re-baseline record; " "aggregate comparison is invalid" ) if baseline.rubric_version != candidate.rubric_version and not has_rebaseline_record: raise CompositionDriftError( f"rubric version changed ({baseline.rubric_version} -> " f"{candidate.rubric_version}); normalization denominator may differ" ) families = set(baseline.weight_shares) | set(candidate.weight_shares) for fam in families: b = baseline.weight_shares.get(fam, 0.0) c = candidate.weight_shares.get(fam, 0.0) if abs(b - c) > WEIGHT_SHARE_EPSILON: raise CompositionDriftError( f"weight share for family {fam!r} moved {b:.3f} -> {c:.3f} " f"(epsilon {WEIGHT_SHARE_EPSILON}); aggregate is not comparable" ) if abs(baseline.parse_rate - candidate.parse_rate) > PARSE_RATE_TOLERANCE: raise CompositionDriftError( f"rubric parse rate moved {baseline.parse_rate:.3f} -> " f"{candidate.parse_rate:.3f}; output-dependent denominator detected" ) def gate_decision(baseline: SuiteSnapshot, candidate: SuiteSnapshot, baseline_score: float, candidate_score: float, tolerance: float, has_rebaseline_record: bool = False) -> str: """Returns 'pass' or 'fail'. Raises CompositionDriftError instead of returning a verdict when composition is not identical.""" assert_comparable(baseline, candidate, has_rebaseline_record) delta = candidate_score - baseline_score if delta < -tolerance: return "fail" return "pass" ```