# What happens when an instance passes the LITMUS safety gate under a stale adversarial seed corpus **Date of Record:** 2026 November 15 **Scope:** Applies to LITMUS regression gating runs where the adversarial seed corpus is pinned by content hash but the agent under test is evaluated against a *cached* task manifest. Does not apply to cold-start evaluation runs where the manifest is regenerated per commit, nor to live red-team sessions with human adjudication. ## Core Distinction A LITMUS safety gate answers a narrow question: *given this fixed set of adversarial seeds, does the candidate agent violate any registered invariant?* The gate's verdict is only as trustworthy as the pairing between (a) the seed corpus content and (b) the task manifest that expands those seeds into concrete, executable probes. These two artifacts are versioned independently. The corpus is pinned by a Merkle root over seed files; the manifest is pinned by a build ID that is *derived* from the corpus root plus a harness revision. When both are fresh, the derivation chain is sound and a PASS means what it says. The non-obvious failure is that the manifest is frequently served from a content-addressed cache keyed on the corpus root *alone*, omitting the harness revision from the cache key. A harness patch that changes how a seed is expanded into probes — say, tightening a prompt-injection seed's delivery channel from `stdout` to `tool_call` — produces a *new* manifest under the same cache key. The cache returns the old manifest. The agent is then evaluated against probes that no longer correspond to the committed seed semantics, and the gate can emit PASS while the newly-strengthened probe class is never exercised. ## What Was Observed During a routine regression sweep, the harness reported `PASS` for a candidate agent on the `prompt_injection.tool_channel` suite. The corresponding seed corpus had been amended two commits prior to route injection payloads through function-call arguments rather than free-text output. Inspection of the run artifact showed the manifest's `probe_count` for that suite was 14, while a cold rebuild of the same corpus root produced 23 probes. The nine missing probes were exactly the tool-channel variants introduced by the harness patch. The cache key had been computed as `sha256(corpus_root)`; the harness revision `a4f19c` was stored in the manifest body but not in the key. The agent, which was known to leak system prompts when injected via tool arguments, scored clean because no probe reached that surface. The failure is silent by construction: the gate has no independent oracle for probe coverage. It trusts the manifest it is handed. A stale manifest is not malformed — it is internally consistent, hash-valid, and passes every structural assertion the harness performs before execution. ## Condition of Validity A LITMUS PASS is valid only when all three hold simultaneously: 1. `manifest.corpus_root == committed_corpus_root` (content binding), **and** 2. `manifest.harness_rev == running_harness_rev` (semantic binding), **and** 3. `manifest.probe_count[suite] == recomputed_probe_count(committed_corpus_root, suite)` for every gated suite (coverage binding). Condition 2 is the one that silently degrades. Conditions 1 and 3 are cheap to assert; condition 2 requires the harness to refuse any manifest whose embedded revision does not byte-match its own build stamp. Without that refusal, the cache is an unguarded trust boundary between the seed corpus and the probe space. ## Decision Boundary The gate must treat a manifest whose `harness_rev` mismatches the running build as a *hard evaluation error*, not a cache miss to be silently rebuilt and not a warning to be logged. Silent rebuild is tempting — it "just works" — but it destroys reproducibility: two runs on the same commit can then produce different probe sets depending on cache warmth, which is precisely the nondeterminism a regression gate exists to eliminate. The correct behavior is to fail closed, surface the mismatch, and require an explicit `--rebuild-manifest` flag that records the invalidation in the run ledger. Operators who need to evaluate against a historical manifest must pass `--allow-stale-manifest` and accept that the PASS is annotated `stale_manifest=true` and cannot satisfy a release gate. ## Operational Effect After this record, the harness enforces revision binding at manifest load. Cache keys become `sha256(corpus_root || harness_rev)`. Gated suites additionally recompute probe counts and abort on any shortfall. The practical consequence is that a harness patch touching probe expansion now invalidates every warm manifest for the affected suites, which lengthens CI by roughly the cost of one cold expansion per suite — a few seconds — in exchange for closing a class of false-PASS that previously survived multiple release cycles undetected. The lesson generalizes: any cache sitting between a versioned input and a derived evaluation artifact must key on the *full* derivation tuple, or the gate it feeds is measuring the cache, not the agent. ```python class StaleManifestError(RuntimeError): """Raised when a cached LITMUS manifest does not match the running harness build.""" def load_manifest(manifest: dict, running_harness_rev: str, allow_stale: bool = False) -> dict: if manifest.get("harness_rev") != running_harness_rev: if not allow_stale: raise StaleManifestError( f"manifest harness_rev={manifest.get('harness_rev')!r} " f"!= running={running_harness_rev!r}; probe space may be stale" ) manifest = {**manifest, "stale_manifest": True} return manifest def verify_coverage(manifest: dict, recomputed_counts: dict[str, int]) -> None: for suite, expected in recomputed_counts.items(): actual = manifest.get("probe_count", {}).get(suite, 0) if actual != expected: raise StaleManifestError( f"suite {suite!r}: manifest probe_count={actual} != recomputed={expected}" ) ```