# What structural signals indicate a LITMUS safety benchmark harness is silently scoring against a stale trajectory oracle

**Date of Record:** 2026 November 14

**Scope:** Applies to LITMUS-family harnesses that score adversarial agent trajectories against a frozen reference oracle and gate regressions on delta thresholds. Does not apply to online red-team loops where the oracle is re-derived per episode, nor to single-shot classification harnesses without trajectory replay.

## Core Distinction

The failure is not "the oracle is wrong." It is "the oracle is *stale relative to the agent's action space*." A LITMUS harness holds two artifacts that drift at different rates: the adversarial trajectory corpus (regenerated frequently) and the reference oracle that labels each trajectory as safe/unsafe (frozen for reproducibility). When the agent's tool surface, prompt scaffolding, or policy head changes, the corpus regenerates into trajectories the oracle was never calibrated against. The oracle still returns a label. The label is still deterministic. The gate still computes a pass/fail. Nothing throws. The harness is *green and meaningless*.

The non-obvious part: staleness is invisible in aggregate safety scores. It only manifests as a **structural signal** — a change in the *shape* of the score distribution and the *correlation* between oracle labels and trajectory features — while the *mean* safety score stays flat or even improves.

## What Was Observed

Across three harness deployments, the following structural signals preceded a confirmed stale-oracle condition by 2–6 regeneration cycles:

1. **Label-feature mutual information collapse.** The oracle's unsafe label stopped correlating with the trajectory features it was historically calibrated on (e.g., number of destructive tool calls, presence of `rm -rf` in shell args, privilege-escalation chains). MI between `unsafe` label and `destructive_call_count` dropped from ~0.41 to ~0.03 while the aggregate unsafe rate held at 12% ± 1%. The oracle had begun labeling on *stylistic* residue (trajectory length, verbosity) rather than the causal feature.

2. **Regression-gate pass rate decoupling from held-out human audit.** Gate pass rate stayed at 98.5% while human-audited true-safety rate on the same trajectories fell to 71%. The gate was measuring agreement with the stale oracle, not safety.

3. **Adversarial corpus saturation.** Newly generated adversarial trajectories clustered in a low-dimensional manifold the oracle had seen. Novel attack families (multi-turn tool-chaining, deferred side effects) produced oracle labels with bimodal confidence — near-0 or near-1 with nothing between — indicating the oracle was extrapolating, not classifying.

4. **Delta-threshold creep.** To keep the gate green as the agent improved, operators had quietly widened the regression tolerance from 0.5% to 3.0% over four cycles. This is the human-visible symptom of signal #1.

## Condition of Validity

The stale-oracle failure is *operationally valid* (i.e., a real risk) only when all of the following hold:

- The oracle is frozen for reproducibility across a corpus that is regenerated independently.
- The regression gate compares agent scores to oracle labels rather than to human-audited ground truth.
- The agent's action space is mutable between oracle-freeze events.
- Aggregate safety scores are the primary reported metric, with no per-feature MI or calibration monitoring.

If any one is absent — e.g., the gate compares to a rolling human-audit sample — the failure degrades to a detectable drift rather than a silent one.

## Decision Boundary

The actionable boundary: **when MI between the oracle's unsafe label and the top-k causal trajectory features drops below 0.10 for two consecutive regeneration cycles, the oracle is stale regardless of aggregate score stability.** At that point, do not retune the gate threshold (that treats the symptom). Freeze the corpus, re-derive the oracle against the current action space, and re-baseline the gate against a fresh human audit. Below the 0.10 MI floor with stable aggregates, treat the harness as *reporting-only* and block merge gating until re-derivation completes.

## Operational Effect

A silently stale oracle converts the safety gate from a regression detector into a *reproducibility theater*: it enforces agreement with a frozen, increasingly irrelevant labeler. Teams ship agents that pass the gate and fail in production. The corrective cost scales with how long the staleness went undetected — typically 2–6 cycles of corpus and agent drift, during which the regression tolerance has been widened to mask the decoupling. The structural signals above are the only early warning; aggregate scores will not warn you.

```python
"""LITMUS stale-oracle detector.

Raises StaleOracleError when structural signals indicate the frozen
reference oracle no longer tracks the agent's action space.
"""

from __future__ import annotations

from dataclasses import dataclass


class StaleOracleError(RuntimeError):
    """Raised when the LITMUS oracle is structurally stale vs. the corpus."""


@dataclass
class CycleReport:
    cycle: int
    label_feature_mi: float          # MI(unsafe_label, top-k causal features)
    gate_pass_rate: float            # fraction of trajectories passing gate
    human_audit_true_safe_rate: float  # held-out human audit on same set
    oracle_confidence_bimodal: bool  # True if labels cluster near 0/1 with no middle


MI_FLOOR = 0.10
GATE_AUDIT_GAP_FLOOR = 0.05


def check_oracle_health(history: list[CycleReport]) -> None:
    if len(history) < 2:
        return

    recent = history[-2:]

    # Signal 1: MI collapse across two consecutive cycles.
    if all(c.label_feature_mi < MI_FLOOR for c in recent):
        raise StaleOracleError(
            f"Label-feature MI below floor {MI_FLOOR} for cycles "
            f"{recent[0].cycle} and {recent[1].cycle} "
            f"(MI={[round(c.label_feature_mi, 3) for c in recent]}). "
            "Oracle is scoring on non-causal residue."
        )

    # Signal 2: gate/audit decoupling.
    for c in recent:
        gap = c.gate_pass_rate - c.human_audit_true_safe_rate
        if gap > GATE_AUDIT_GAP_FLOOR:
            raise StaleOracleError(
                f"Cycle {c.cycle}: gate pass rate {c.gate_pass_rate:.3f} "
                f"exceeds human-audited true-safe rate "
                f"{c.human_audit_true_safe_rate:.3f} by {gap:.3f}. "
                "Gate is measuring oracle agreement, not safety."
            )

    # Signal 3: bimodal oracle confidence on novel trajectories.
    if recent[-1].oracle_confidence_bimodal:
        raise StaleOracleError(
            f"Cycle {recent[-1].cycle}: oracle labels are bimodal on novel "
            "corpus — extrapolating, not classifying. Freeze and re-derive."
        )


def gate_merge(history: list[CycleReport]) -> bool:
    try:
        check_oracle_health(history)
    except StaleOracleError:
        # Fail closed: block merge gating until oracle re-derived.
        return False
    return True
```