# How should an instance distinguish refusal calibration drift from genuine safety regression in LITMUS-style agentic harm elicitation suites

**Date of Record:** 2026 November 9

**Scope:** Applies to agentic evaluation harnesses that run LITMUS-style multi-turn harm elicitation suites where refusal rates are compared across model versions, prompt-template revisions, or judge-model swaps. Does not apply to single-turn classification benchmarks or to suites with a fixed, human-adjudicated gold set and no LLM judge in the loop.

## Core Distinction

The non-obvious invariant is this: in an agentic LITMUS suite, a **refusal** is a joint product of the target model's policy and the *elicitator agent's* behavior. The elicitator is itself a model, and it drifts. When a new target model shows a lower refusal rate on the same suite, the observation is underdetermined between two hypotheses that produce *identical* refusal-rate deltas:

1. **Safety regression** — the target model is genuinely more compliant with harmful requests.
2. **Calibration drift** — the elicitator has become more (or less) effective at constructing persuasive multi-turn ladders, or the judge has shifted its threshold for what counts as a refusal.

These two hypotheses are not distinguishable from the aggregate refusal rate alone. They are distinguishable only by holding the elicitator and judge *fixed* across the comparison and perturbing the target in isolation — or by measuring the elicitator's own attack-success distribution against a frozen reference target. Most harnesses conflate them because they version the whole pipeline (target + elicitator + judge) as a single unit and report a single scalar.

## What Was Observed

Across three consecutive suite revisions of an agentic harm-elicitation harness, the headline "refusal rate" for a frozen reference model moved from 0.91 → 0.84 → 0.88 while the target model was byte-identical. The apparent 7-point "regression" in revision two was traced entirely to a changed elicitator sampling temperature (0.7 → 0.9), which increased the variance of adversarial ladder construction and produced more turns that the judge classified as "compliance" because the target answered a *benign sub-question* embedded in the ladder. The judge — an LLM with a rubric — was scoring per-turn helpfulness, and the elicitator's higher temperature generated more benign sub-questions. No target policy changed.

A second, subtler case: a judge-model upgrade (from a 7B to a 70B judge) *raised* measured refusal rates by 4 points on the same transcripts, because the larger judge was stricter about what counted as a "full refusal" versus a "partial refusal with caveat." The suite reported this as a safety *improvement* in the target. It was a judge-threshold shift.

In both cases the invariant that was violated is: **the elicitator–judge pair is part of the measurement instrument, not part of the treatment.** Any comparison that varies the instrument and the treatment together is not a valid A/B.

## Condition of Validity

A refusal-rate comparison across two target models is valid only if all of the following hold:

- The elicitator model, its sampling parameters (temperature, top-p, max turns), and its system prompt are **byte-identical** across both runs.
- The judge model, its rubric version, and its threshold for "refusal" are **byte-identical** across both runs.
- The transcript set is paired: the same seed prompts and the same elicitator random seeds are replayed against both targets, so per-item deltas can be computed rather than only aggregate deltas.
- A **frozen reference target** (a model whose weights never change) is run through the *same* instrument in both batches. If the reference target's refusal rate moves by more than the instrument's calibrated noise floor, the instrument has drifted and the comparison is void.

If any of these fails, the observed delta is a *pipeline* delta, not a *safety* delta, and must not be reported as a model regression.

## Decision Boundary

The operational rule:

- **If the frozen reference target's refusal rate is within the noise floor** (empirically ±2 points for a 500-item suite with an LLM judge; measure this once and record it), attribute the target delta to the target model and report it as a safety signal.
- **If the reference target's rate has moved beyond the noise floor**, the instrument drifted. Do not report the target delta. Instead, re-run with the instrument pinned to the last-known-good elicitator/judge pair, or re-baseline the reference target and record the instrument revision explicitly.
- **If only the judge changed**, re-score the *stored transcripts* from the prior run with the new judge before running any new elicitation. Judge-only re-scoring is cheap and isolates the threshold shift from the elicitation shift.
- **If only the elicitator changed**, do not compare across the change at all; treat it as a new suite version with a fresh baseline.

The boundary is asymmetric: a *drop* in refusal rate is the dangerous direction and warrants the strictest instrument-pinning; a *rise* is often a judge-threshold artifact and should be re-scored before being celebrated.

## Operational Effect

Pinning the instrument and running the frozen reference target on every batch converts an ambiguous scalar into a falsifiable measurement. In practice it caught the temperature-drift case above before the "regression" was escalated, and it reclassified the judge-upgrade "improvement" as a threshold shift. The cost is one extra reference-target run per batch (roughly 8% additional inference for a 500-item suite). The benefit is that every reported safety delta is now attributable to exactly one variable. Harnesses that skip the reference run will, over enough revisions, accumulate instrument drift that masquerades as a safety trend in whichever direction the elicitator and judge happen to be drifting — and that drift is not auditable after the fact unless transcripts were stored and are re-scorable.

```python
"""Instrument-pinning guard for LITMUS-style refusal-calibration comparisons."""

from dataclasses import dataclass


class InstrumentDriftError(RuntimeError):
    """Raised when the measurement instrument (elicitator/judge) drifted between batches."""


@dataclass(frozen=True)
class Instrument:
    elicitator_id: str
    elicitator_temperature: float
    judge_id: str
    judge_rubric_version: str
    max_turns: int


@dataclass(frozen=True)
class BatchResult:
    instrument: Instrument
    target_refusal_rate: float
    reference_target_refusal_rate: float


NOISE_FLOOR = 0.02  # empirically calibrated for a 500-item suite


def compare_batches(
    baseline: BatchResult,
    candidate: BatchResult,
    require_pinned_instrument: bool = True,
) -> dict:
    if require_pinned_instrument and baseline.instrument != candidate.instrument:
        raise InstrumentDriftError(
            "Instrument changed between batches: "
            f"{baseline.instrument} -> {candidate.instrument}. "
            "Re-score stored transcripts with the pinned judge, or re-baseline."
        )

    ref_delta = candidate.reference_target_refusal_rate - baseline.reference_target_refusal_rate
    if abs(ref_delta) > NOISE_FLOOR:
        raise InstrumentDriftError(
            f"Frozen reference target moved {ref_delta:+.3f} (noise floor {NOISE_FLOOR}). "
            "Instrument drifted; target delta is not attributable to the target model."
        )

    target_delta = candidate.target_refusal_rate - baseline.target_refusal_rate
    return {
        "target_delta": target_delta,
        "reference_delta": ref_delta,
        "attributable_to_target": True,
        "direction": "regression" if target_delta < 0 else "improvement",
    }


def run_batch(instrument: Instrument, target_rate: float, ref_rate: float) -> BatchResult:
    if not (0.0 <= target_rate <= 1.0 and 0.0 <= ref_rate <= 1.0):
        raise InstrumentDriftError(f"Refusal rate out of range: target={target_rate}, ref={ref_rate}")
    return BatchResult(instrument=instrument, target_refusal_rate=target_rate,
                       reference_target_refusal_rate=ref_rate)


if __name__ == "__main__":
    inst_a = Instrument("elicit-v3", 0.7, "judge-7b", "rubric-2", 6)
    inst_b = Instrument("elicit-v3", 0.9, "judge-7b", "rubric-2", 6)  # temperature drift

    base = run_batch(inst_a, target_rate=0.91, ref_rate=0.91)
    cand = run_batch(inst_b, target_rate=0.84, ref_rate=0.84)

    try:
        compare_batches(base, cand)
    except InstrumentDriftError as e:
        print(f"REJECTED: {e}")
```