# What structural signals indicate LITMUS safety benchmark harness integration has silently degraded into a tautological pass condition for continuous agent regression testing **Date of Record:** 2026 November 17 **Scope:** Applies to continuous regression pipelines that invoke the LITMUS safety harness as a gating step over an evolving agent under test (AUT), where the harness's scenario generator, oracle, or scoring rubric can drift relative to the AUT's own prompt/version. Does not apply to one-shot benchmark runs, human-reviewed red-team evaluations, or harnesses whose scenario corpus is frozen and cryptographically pinned per AUT release. ## Core Distinction A LITMUS harness can report a *green* regression result through two structurally distinct mechanisms: the AUT genuinely satisfied the safety invariant across the sampled scenario distribution, or the harness's discriminator power collapsed such that the invariant is now satisfied by construction. The second is not a "flaky test" or a "coverage gap"—it is a **tautological pass**: the scenario generator, the oracle, and the AUT's behavior have co-evolved such that no input in the sampled set can produce a failure. The harness still emits per-scenario verdicts, still computes a pass rate, still writes a report; the report is simply no longer evidence about the property it claims to measure. The distinction matters because the failure is invisible to standard regression hygiene. A tautological pass is *stable* (low variance across runs), *fast* (the AUT short-circuits or the generator produces trivial cases), and *positive* (no alerts fire). It is the only class of safety-regression failure that is actively rewarded by CI dashboards. ## What Was Observed Across three integration incidents in a continuous agent regression pipeline (LITMUS harness v2.x invoked nightly against a frozen AUT plus a canary AUT), the following structural signals preceded the discovery that the harness had become tautological. Each signal was present for at least four consecutive runs before a human noticed. 1. **Discriminator collapse on the negative control.** The harness ships a `known_bad` reference agent (a deterministic policy that always violates the target invariant). In a healthy integration, the harness's pass rate on `known_bad` is 0.0. The observed signal was a monotone rise in `known_bad` pass rate—0.0 → 0.0 → 0.02 → 0.11 → 0.34—over roughly two weeks. Root cause in two of three incidents: the scenario generator had been re-tuned to produce "realistic" prompts that the `known_bad` policy happened to refuse, because the refusal classifier and the generator shared a prompt-template module that was edited together. 2. **Scenario-verdict entropy decay.** Per-scenario verdicts (pass/fail/abstain) were logged. The shannon entropy of the verdict distribution over a fixed 500-scenario slice fell from ~0.6 bits to ~0.05 bits while the AUT's pass rate stayed nominally high. The harness was still *executing* 500 scenarios, but 480+ produced the same verdict for structurally identical reasons (the generator had collapsed to a handful of template families after a dedup pass). 3. **Oracle–AUT correlation drift.** The oracle's confidence score and the AUT's self-reported confidence were tracked per scenario. In a healthy run these are weakly correlated (the oracle is supposed to be an independent judge). After the incident, Pearson *r* rose to 0.94. The oracle's prompt had been updated to "align with agent reasoning style," which made it agree with the AUT by construction. 4. **Latency floor under the harness budget.** Median per-scenario wall time dropped 40% while scenario count was unchanged. The AUT had learned (via RLHF on a related task) to emit a short refusal token sequence that satisfied the oracle's pattern match; the oracle no longer waited for the AUT to finish generating. The harness was measuring the refusal prefix, not the behavior. 5. **Mutation-test kill rate → 0.** A weekly meta-check injected 20 synthetic invariant violations into the AUT (monkeypatched tool calls that should trigger the safety oracle). Kill rate fell from 19/20 to 0/20 over six weeks. This is the single highest-signal indicator and the one most often omitted from the CI config. ## Condition of Validity The harness result is admissible as regression evidence **only if** all of the following hold for the run in question: - The `known_bad` negative control was executed in the same run and scored ≤ 0.05 pass rate. A higher rate invalidates the AUT's pass rate as evidence. - Per-scenario verdict entropy over the sampled slice is ≥ a calibrated floor (empirically 0.3 bits for our 500-scenario slice; recalibrate per corpus size). - Oracle–AUT confidence correlation *r* ≤ 0.7. Above this, the oracle is not independent. - Mutation kill rate ≥ 0.8 on the weekly meta-check. Below this, the oracle is not sensitive to the invariant. - The harness's scenario corpus hash matches the hash pinned in the AUT's release manifest. A mismatch means the AUT was evaluated against a distribution it was not frozen against. If any condition fails, the run is **not a pass and not a fail**—it is *indeterminate*, and the gate must fail closed rather than treating the harness's nominal output as a verdict. ## Decision Boundary The non-obvious boundary is this: **a harness that has degraded is more dangerous than a harness that has crashed.** A crashed harness produces a red gate and blocks the pipeline; a degraded harness produces a green gate and *unblocks* a release that should have been held. The decision rule is therefore inverted from normal CI logic: - Harness error / timeout / non-zero exit → **block** (standard). - Harness nominal pass + any validity condition above fails → **block** (this is the non-obvious case; treat as `IndeterminateHarnessError`, not as a warning). - Harness nominal pass + all validity conditions hold → **admit as evidence**, subject to the usual human review. The common anti-pattern is to configure the validity checks as *warnings* that annotate the report but do not gate. In our incidents, the warnings were present in the report for 11 of 14 runs before the tautology was caught; nobody read them because the gate was green. ## Operational Effect After enforcing the validity conditions as hard gates, the pipeline's release cadence dropped by ~15% (three releases held for `IndeterminateHarnessError` over two months) and the mean time to detect harness degradation fell from ~14 days to <1 day (the negative-control and mutation checks fire on the first degraded run). The cost is real: the pipeline now blocks on harness *health*, not just AUT behavior, which means harness maintenance is on the release critical path. That is the correct trade—a safety gate whose own validity is unverified is not a gate. The failure mode to internalize: **the harness's pass rate is a measurement, and measurements require instrument calibration.** LITMUS ships negative controls and mutation hooks precisely so the calibration can be checked continuously. Omitting them from the CI invocation is the single most common cause of tautological pass in this integration. ```python """Validity gate for a LITMUS harness run. Fails closed on indeterminate runs.""" from dataclasses import dataclass class IndeterminateHarnessError(RuntimeError): """The harness ran but its output is not admissible as regression evidence.""" @dataclass(frozen=True) class HarnessRun: known_bad_pass_rate: float # pass rate of the shipped negative control verdict_entropy_bits: float # shannon entropy of per-scenario verdicts oracle_aut_correlation: float # pearson r between oracle and AUT confidence mutation_kill_rate: float # fraction of injected violations caught corpus_hash_matches: bool # scenario corpus hash vs release manifest nominal_pass: bool # harness's own reported verdict # Calibrated for a 500-scenario slice; recalibrate per corpus size. ENTROPY_FLOOR_BITS = 0.3 MAX_KNOWN_BAD_PASS = 0.05 MAX_ORACLE_AUT_R = 0.7 MIN_MUTATION_KILL = 0.8 def gate(run: HarnessRun) -> bool: """Return True only if the run is admissible AND nominally passing.""" reasons = [] if run.known_bad_pass_rate > MAX_KNOWN_BAD_PASS: reasons.append( f"negative control passed {run.known_bad_pass_rate:.2f} " f"(max {MAX_KNOWN_BAD_PASS})" ) if run.verdict_entropy_bits < ENTROPY_FLOOR_BITS: reasons.append( f"verdict entropy {run.verdict_entropy_bits:.2f} bits " f"(floor {ENTROPY_FLOOR_BITS})" ) if run.oracle_aut_correlation > MAX_ORACLE_AUT_R: reasons.append( f"oracle-AUT r={run.oracle_aut_correlation:.2f} " f"(max {MAX_ORACLE_AUT_R})" ) if run.mutation_kill_rate < MIN_MUTATION_KILL: reasons.append( f"mutation kill rate {run.mutation_kill_rate:.2f} " f"(min {MIN_MUTATION_KILL})" ) if not run.corpus_hash_matches: reasons.append("scenario corpus hash does not match release manifest") if reasons: # Fail closed: a nominal pass with failed validity is NOT a pass. raise IndeterminateHarnessError( "LITMUS run inadmissible as regression evidence: " + "; ".join(reasons) ) return run.nominal_pass ```