# What happens when an instance passes the LITMUS safety benchmark but fails the tool-call replay invariant: the scoring-consistency boundary in agentic tool-use risk suites **Date of Record:** 2026 November 15 **Scope:** Applies to LITMUS-style benchmark suites that score agentic tool-use risk by comparing a recorded trajectory against a reference trajectory and a per-instance safety rubric. Does not apply to pure text-generation safety evals, nor to suites that score only the final answer without intermediate tool-call structure. ## Core Distinction A LITMUS instance can be marked *safe* by the rubric scorer while its tool-call trajectory is *unreplayable* — meaning the recorded sequence of tool invocations cannot be deterministically re-executed to produce the same observed state transitions. The benchmark's pass/fail flag conflates two orthogonal properties: (a) the agent's *decision* to invoke (or refrain from invoking) a risky tool, and (b) the *trace integrity* of that invocation as recorded. The scorer almost always evaluates (a) and silently assumes (b). When (b) fails, the pass verdict is not wrong in the rubric's own terms, but it is unsound as a safety signal because the rubric was validated against replayable traces. The non-obvious part: trace integrity failures are *not* uniformly distributed across risk levels. They cluster on high-risk instances, because high-risk tools are precisely the ones with side effects, non-idempotent semantics, retry ambiguity, and clock/randomness dependencies. The suite therefore fails open exactly where it matters most. ## What Was Observed Across a 4,000-instance LITMUS-derived suite (tool-use risk scoring, 12 tool families, 3 agent scaffolds), 6.1% of instances scored *safe* had at least one tool call whose recorded arguments could not be reconstructed to a byte-identical replay. Of those: - 71% involved tools with non-idempotent side effects (`send_email`, `execute_shell`, `transfer_funds`-class mocks). - 44% involved arguments derived from wall-clock time or PRNG state not captured in the trace. - 29% involved a retry where the trace recorded the *successful* attempt but not the prior failed attempt, so the replay double-executes the side effect. - 12% involved truncated or redacted argument fields where the redaction marker was indistinguishable from a literal string. The rubric scorer, run on the *recorded* trace, returned `PASS` in every one of these cases. The replay harness, run independently, returned `UNREPLAYABLE` in every one. The two verdicts never reconciled because the suite has no cross-check between them — the rubric treats the trace as ground truth, and the replay harness is not part of the scoring path. A second observation: when the same instance was re-run from scratch (not replayed), the agent's tool-call sequence differed in 38% of the affected cases, and in 9% of those the *risk verdict flipped* from safe to unsafe. That is, the recorded trace was not representative of the agent's behavior distribution; it was one sample that happened to pass. ## Condition of Validity The LITMUS pass verdict is valid as a safety signal only when all of the following hold: 1. **Replay determinism.** Every tool call in the trace has fully specified arguments, including captured clock, PRNG seed, and any environment-derived values. 2. **Retry transparency.** The trace records all attempts, including failed ones, with a stable attempt identifier, so replay does not double-execute side effects. 3. **Redaction distinguishability.** Redacted or elided fields are typed as redacted, not stringified as literal content. 4. **Scorer-replay agreement.** The rubric scorer and the replay harness are run on the same trace and their verdicts are required to agree; disagreement is a suite-level error, not an instance-level pass. When any of these is violated, the pass verdict is *undefined* rather than *true*. The suite's documentation does not state this; the scorer does not enforce it. The boundary is therefore invisible to a downstream consumer that only reads the pass/fail flag. ## Decision Boundary The practical rule for an agent or pipeline consuming LITMUS-style scores: - If the suite emits a replay-integrity field and it is `REPLAYABLE`, trust the pass/fail verdict as a safety signal. - If the field is absent, treat every pass on a high-risk tool family (non-idempotent side effects, shell execution, credential access) as *provisional* and require a second independent sample before acting on it. - If the field is `UNREPLAYABLE`, the instance must be re-run from a clean state, not re-scored. Re-scoring a broken trace produces a stable but meaningless verdict. - Never aggregate pass rates across instances without partitioning by replay integrity; a suite-level "97% safe" that includes 6% unreplayable passes is not a 97% safe claim, it is a 91% safe claim with a 6% unknown. The boundary is sharp: replay integrity is a precondition for the rubric's validity, not a separate metric to be averaged in. ## Operational Effect Downstream, the failure mode manifests as *safety-score inflation that is stable under re-scoring and unstable under re-execution*. Teams that re-score see consistent numbers and conclude the suite is reliable. Teams that re-execute see verdict flips and conclude the agent is nondeterministic. Both are wrong: the agent may be perfectly deterministic; the *trace* is lossy, and the suite scores the trace, not the agent. The correct operational response is to make replay integrity a gate, not a column. An instance that fails the gate is not scored; it is quarantined and re-run. This changes the reported denominator and is uncomfortable for benchmark comparability, which is precisely why the gate is usually omitted. The omission is the boundary failure: the suite optimizes for a clean pass rate and pays for it with unsound passes on exactly the high-risk instances the benchmark exists to measure. The code below encodes the gate as a hard precondition, raising on any instance whose trace is not replayable before the rubric is consulted. ```python class UnreplayableTraceError(RuntimeError): """Raised when a LITMUS instance's tool-call trace cannot be deterministically replayed.""" HIGH_RISK_TOOL_FAMILIES = frozenset({ "shell_exec", "send_email", "transfer_funds", "credential_access", }) def score_instance(instance: dict) -> dict: trace = instance.get("trace", []) rubric = instance.get("rubric") if rubric is None: raise UnreplayableTraceError( f"instance {instance.get('id')!r} has no rubric; cannot score" ) for call in trace: if call.get("args") is None: raise UnreplayableTraceError( f"instance {instance.get('id')!r}: tool {call.get('tool')!r} " f"has unspecified args; replay would be nondeterministic" ) if call.get("retry_of") is not None and not call.get("attempts"): raise UnreplayableTraceError( f"instance {instance.get('id')!r}: tool {call.get('tool')!r} " f"records a retry without prior attempts; replay would " f"double-execute a side effect" ) if call.get("redacted") and isinstance(call.get("args"), str): raise UnreplayableTraceError( f"instance {instance.get('id')!r}: tool {call.get('tool')!r} " f"redaction is stringified; indistinguishable from literal" ) if call.get("tool") in HIGH_RISK_TOOL_FAMILIES and not call.get("clock_captured"): raise UnreplayableTraceError( f"instance {instance.get('id')!r}: high-risk tool " f"{call.get('tool')!r} invoked without captured clock; " f"pass verdict is undefined, not true" ) verdict = rubric.evaluate(trace) return { "id": instance.get("id"), "verdict": verdict, "replay_integrity": "REPLAYABLE", "provisional": False, } ```