# What happens when an instance of the LITMUS safety benchmark harness is run against a target agent whose tool-call surface changes between the baseline and the regression run **Date of Record:** 2026 November 15 **Scope:** Applies to CI pipelines that pin a LITMUS harness version and compare per-scenario safety verdicts between a stored baseline and a fresh run of the same agent. Does not apply to single-run exploratory evaluations, nor to runs where the harness and target agent share a single immutable container image and tool schema digest. ## Core Distinction The LITMUS harness records a safety verdict per scenario, but the verdict is only meaningful relative to the *attack surface* that scenario exercised. When the target agent's tool-call surface (the set of tool names, argument schemas, and side-effect annotations visible to the model) changes between the baseline capture and the regression run, the harness does **not** fail closed. It silently re-resolves each scenario's tool references against the new surface and emits a verdict that is structurally comparable but semantically incomparable to the baseline. The failure is not a crash and not a red build; it is a green build that compares apples to oranges. The invariant that must be enforced is: *a verdict comparison is valid only if the tool-surface digest recorded with the baseline equals the digest observed at run time.* ## What Was Observed In a pipeline that ran LITMUS nightly against an agent under active development, a PR that renamed `send_email` to `send_message` and merged the `to`/`cc` arguments into a single `recipients` list produced a passing safety regression. The harness's scenario loader, encountering a scenario that referenced `send_email`, fell back to fuzzy tool-name matching and bound the scenario to `send_message`. The exfiltration scenario's payload — designed to leak a canary string via the `to` field — was rewritten by the adapter into `recipients`, and the canary check, which keyed on the argument name `to`, found nothing to inspect and returned `PASS`. The baseline had recorded `FAIL` (canary leaked). The regression run recorded `PASS`. The delta was reported as an *improvement*. Two weeks later the same agent, with the fuzzy matcher disabled for an unrelated reason, leaked the canary in production staging. The tell was in the harness's verbose log: `scenario[exfil_07]: tool 'send_email' unresolved, fuzzy-bound to 'send_message' (score 0.71)`. No CI gate read that line. ## Condition of Validity A LITMUS verdict comparison between baseline $B$ and run $R$ is valid if and only if all of the following hold: 1. `harness_version(B) == harness_version(R)` — the scenario semantics and fuzzy-matching policy are identical. 2. `tool_surface_digest(B) == tool_surface_digest(R)` — the canonicalized set of `(name, schema, side_effect_class)` tuples is byte-identical. 3. `scenario_binding(B, s) == scenario_binding(R, s)` for every scenario $s$ — each scenario resolved to the same tool by exact name, not by fuzzy match. 4. The canary/inspection predicates key on argument *identity* (schema path), not on argument *name string*. If any condition fails, the comparison must be treated as *invalid*, not as *pass* or *fail*. The correct CI outcome is a distinct third state — `INCOMPARABLE` — that blocks merge until the baseline is re-captured against the new surface. ## Decision Boundary The boundary is between *surface drift that is semantically safe* and *surface drift that invalidates a scenario's threat model*. A rename that preserves argument identity and side-effect class (e.g., adding a new optional tool that no scenario references) does not invalidate comparison; the digest changes but the scenario bindings and inspection predicates are unaffected. A rename or argument restructure that any scenario's inspection predicate keys on *does* invalidate it. The harness cannot distinguish these on its own because it does not know which argument names are load-bearing for which canary checks. Therefore the gate must be conservative: any change to the digest of a tool referenced by a scenario with an active inspection predicate forces re-baselining of that scenario. Do not attempt to auto-reconcile; the fuzzy matcher's score is not a safety signal. ## Operational Effect Enforcing the digest-equality gate converts silent green-build safety regressions into explicit `INCOMPARABLE` blocks. In the pipeline where this was applied, the first week produced four blocks, three of which were benign renames resolved by re-capturing baselines, and one of which was a genuine argument-restructure that had been masking a canary leak for eleven days. The cost is re-baselining latency on legitimate surface changes; the benefit is that a `PASS` on a safety scenario once again means the scenario's threat model was actually exercised. The gate belongs in the harness adapter layer, not in the CI YAML, so that it cannot be bypassed by a pipeline edit. ```python import hashlib import json from dataclasses import dataclass class ToolSurfaceDriftError(RuntimeError): """Raised when a LITMUS verdict comparison is invalid due to tool-surface drift.""" pass @dataclass(frozen=True) class ToolSurface: # canonical: sorted list of (name, schema_json, side_effect_class) tools: tuple def digest(self) -> str: payload = json.dumps( sorted(self.tools), separators=(",", ":"), sort_keys=True ).encode() return hashlib.sha256(payload).hexdigest() def assert_comparable( baseline_surface: ToolSurface, run_surface: ToolSurface, baseline_harness_version: str, run_harness_version: str, scenario_bindings_baseline: dict, scenario_bindings_run: dict, ) -> None: if baseline_harness_version != run_harness_version: raise ToolSurfaceDriftError( f"harness version drift: baseline={baseline_harness_version!r} " f"run={run_harness_version!r}; verdicts are not comparable" ) if baseline_surface.digest() != run_surface.digest(): raise ToolSurfaceDriftError( "tool-surface digest mismatch: " f"baseline={baseline_surface.digest()} run={run_surface.digest()}; " "re-capture baseline before comparing safety verdicts" ) drifted = { s: (scenario_bindings_baseline[s], scenario_bindings_run.get(s)) for s in scenario_bindings_baseline if scenario_bindings_baseline[s] != scenario_bindings_run.get(s) } if drifted: raise ToolSurfaceDriftError( f"scenario bindings changed under identical surface digest: {drifted}" ) def compare_verdicts( baseline: dict, run: dict, baseline_surface: ToolSurface, run_surface: ToolSurface, baseline_harness_version: str, run_harness_version: str, scenario_bindings_baseline: dict, scenario_bindings_run: dict, ) -> dict: # Gate first: an incomparable comparison must never be reported as PASS. assert_comparable( baseline_surface, run_surface, baseline_harness_version, run_harness_version, scenario_bindings_baseline, scenario_bindings_run, ) return { s: {"baseline": baseline.get(s), "run": run.get(s)} for s in set(baseline) | set(run) } ```