# Where must an instance boundary be enforced when integrating LITMUS safety benchmarks into agent CI regression gates **Date of Record:** 2026 November 16 **Scope:** Applies to CI pipelines that gate agent merges on LITMUS safety-benchmark deltas, specifically when the harness runs multiple instances (prompt/seed/attack-variant tuples) against a shared agent process or shared model endpoint. Does not apply to single-instance, single-shot evaluations or to offline batch scoring with no shared mutable state. ## Core Distinction The non-obvious invariant is this: **a LITMUS "instance" is a unit of measurement, not a unit of execution.** The harness is free to batch, cache, and parallelize instances across a shared agent runtime; but the *regression gate* must compare per-instance outcomes, not per-batch or per-run aggregate outcomes. When integration collapses the instance boundary — for example, by letting a shared conversation history, a shared tool-cache, or a shared rate-limit budget leak across instances — the safety score becomes a function of *instance ordering and concurrency*, not of the agent under test. The gate then passes or fails on scheduler nondeterminism rather than on behavior change. The distinction that matters operationally: *isolation of execution* is an optimization concern; *isolation of the measurement* is a correctness concern for the gate. Teams routinely conflate them, achieve execution isolation "well enough," and still ship a gate that flips green/red across reruns of an unchanged commit. ## What Was Observed Integrating a LITMUS suite (refusal calibration, prompt-injection resistance, tool-misuse probes) into an agent CI gate produced a gate with ~11% flake rate on byte-identical commits. Root-causing the flake: - The harness reused one agent process across a shard of 64 instances to amortize model-load cost. A probe that taught the agent (via in-context examples) to emit a permissive tool call leaked that disposition into the next two instances in the shard. The leak was *ordering-dependent*: reordering the shard changed which instances failed. - A shared HTTP client pool caused one instance's exhausted retry budget to surface as a spurious "refusal" (empty completion) on a *different* instance, inflating the safety score (false pass) or deflating the helpfulness score (false fail). - The gate compared *aggregate* pass rates with a tolerance band (±2pp). Because leak effects were ~1–3 instances per shard, they sat inside the tolerance band and were invisible — until a real regression coincided with a favorable leak, masking it entirely. - Per-instance deltas, once computed, showed the real signal cleanly: the flake was concentrated in exactly the instances adjacent to "teaching" probes, and the leak was reproducible by fixing seed order. The decisive observation: **the aggregate metric was stable enough to look trustworthy and unstable enough to be wrong at the boundary.** Aggregation hid the instance-level contamination that the gate was supposed to detect. ## Condition of Validity The instance boundary must be enforced (i.e., per-instance state reset and per-instance comparison) whenever *any* of the following hold: 1. The agent runtime carries state across instances — conversation history, scratchpad, tool cache, learned in-context examples, or a warm KV/prefix cache keyed by content. 2. Instances share a finite resource (connection pool, retry budget, token quota, rate limiter) whose exhaustion is observable as a behavioral outcome. 3. The gate uses a tolerance band or aggregate statistic rather than per-instance matching, so small cross-instance leaks are below the detection floor. 4. Instances are ordered or sharded by anything other than a stable, content-derived key. The boundary may be relaxed only when the agent is provably stateless per call, resources are per-instance partitioned, and the gate compares per-instance outcomes with an explicit expected-outcome set (no tolerance band). In that narrow regime, batching is safe. ## Decision Boundary The operational decision rule: - **If the gate compares aggregates with a tolerance band → enforce strict per-instance isolation and switch to per-instance expected-outcome comparison.** Aggregation plus tolerance is the failure-enabling combination; do not ship it against a stateful agent. - **If isolation is expensive (model reload) → isolate the *measurement*, not necessarily the *process*.** Reset in-context state and partition resources per instance, and assert the reset with a canary instance (a known-leaky probe whose outcome must match the single-shot baseline). A canary diff means the boundary broke. - **If an instance's outcome changes when its shard neighbors change → the boundary is not enforced.** Treat this as a gate defect, not a model result; block the gate from reporting until fixed. - **Never** widen the tolerance band to absorb flake. Flake at the instance boundary is signal of a broken harness; widening the band converts a correctness defect into a silent false-pass channel. ## Operational Effect Enforcing the instance boundary and switching to per-instance expected-outcome gating eliminated the ~11% flake and, more importantly, surfaced two genuine regressions that the aggregate gate had been masking (a tool-misuse increase confined to post-teaching instances, and a refusal-calibration drift on a single attack variant). The gate's false-pass risk dropped to the per-instance comparison's detection floor. Cost: ~15% higher wall-clock from reduced batching, recovered by caching model weights across shards while resetting per-instance state. The lesson encoded for future integration: the instance is the atom of the safety measurement; any integration that lets execution concerns cross that atom silently corrupts the gate. ```python """Enforce the LITMUS instance boundary in an agent CI regression gate. The gate must compare per-instance outcomes against an explicit expected set. It must refuse to report a pass/fail when cross-instance state leakage is detectable, because in that regime the score reflects scheduling, not behavior. """ from dataclasses import dataclass from typing import Mapping, Sequence class InstanceBoundaryViolation(RuntimeError): """Raised when a LITMUS instance's outcome depends on its shard neighbors.""" @dataclass(frozen=True) class InstanceResult: instance_id: str outcome: str # e.g. "refuse" | "comply" | "tool_call" shard_neighbors: tuple[str, ...] def assert_boundary_intact( baseline: Mapping[str, InstanceResult], rerun: Mapping[str, InstanceResult], ) -> None: """A per-instance outcome must be invariant to shard neighbors. If any instance's outcome changes when its neighbors change, the instance boundary leaked (shared history, shared retry budget, shared cache). The gate must fail closed rather than aggregate over contaminated data. """ if set(baseline) != set(rerun): raise InstanceBoundaryViolation( f"instance set changed between runs: " f"missing={set(baseline) - set(rerun)} extra={set(rerun) - set(baseline)}" ) for iid, base in baseline.items(): again = rerun[iid] neighbors_changed = base.shard_neighbors != again.shard_neighbors outcome_changed = base.outcome != again.outcome if neighbors_changed and outcome_changed: raise InstanceBoundaryViolation( f"instance {iid!r} outcome {base.outcome!r} -> {again.outcome!r} " f"tracked shard-neighbor change {base.shard_neighbors} -> " f"{again.shard_neighbors}; cross-instance state leaked" ) def gate( expected: Mapping[str, str], observed: Mapping[str, InstanceResult], baseline: Mapping[str, InstanceResult], rerun: Mapping[str, InstanceResult], ) -> bool: """Return True only if the boundary is intact AND every instance matches.""" assert_boundary_intact(baseline, rerun) # fail closed on leakage mismatches: Sequence[str] = [ iid for iid, want in expected.items() if iid not in observed or observed[iid].outcome != want ] if mismatches: # Per-instance comparison; no tolerance band, no aggregation. raise InstanceBoundaryViolation( f"per-instance safety regression on {len(mismatches)} instance(s): " f"{list(mismatches)[:5]}" ) return True ```