# What structural signals indicate a LITMUS safety benchmark harness has silently desynchronized from the agent under test in CI

**Date of Record:** 2026 November 11

**Scope:** Applies to continuous red-teaming pipelines where a LITMUS-style safety harness drives an agent through adversarial probe suites inside CI and gates merges on aggregate safety scores. Does not apply to one-shot offline evaluations, human-in-the-loop red teams, or harnesses that re-instantiate the agent from a pinned artifact per probe.

## Core Distinction

The failure that matters is not a failing safety score. A failing score is signal: the harness observed the agent and the agent behaved unsafely. The boundary failure is **harness–agent desynchronization**, where the harness reports a *passing* (or stably unchanged) score while no longer exercising the agent that will actually ship. Desynchronization is silent because it produces exactly the output a healthy pipeline produces: green, monotonic, low-variance scores.

Two structural causes dominate. First, **probe-suite/agent-version skew**: the harness pins its probe corpus and its expected-response fixtures to a model or prompt revision, while the agent under test is rebuilt from `HEAD` on every CI run. Second, **stateful probe contamination**: probes that assume a fresh agent session leak conversational or tool state into subsequent probes, so a probe that would trigger an unsafe behavior is evaluated against an agent already "warmed" into a refusal posture by an earlier probe. Both yield the same observable: the harness is measuring a *different* system than the one under test.

## What Was Observed

Across several CI integrations, the following structural signals preceded a confirmed desynchronization event. None of them is a score change; all of them are properties of the harness run itself.

1. **Probe-to-agent-version binding is absent from the artifact.** The harness emits a report keyed by probe-suite hash but not by the agent build digest. When the agent binary/prompt bundle changes, the report hash is identical to the previous run. A stable report hash across a changed agent commit is the single strongest indicator that the harness is not coupled to the artifact under test.

2. **Refusal-rate plateau at a suspiciously round value.** Desynchronized runs converge on refusal rates that are stable to within a fraction of a percent across unrelated commits — e.g., exactly 100% refusal on a category, or a rate identical to the harness's own fixture baseline. Real agents under adversarial probing show variance; a fixture is being echoed, not an agent being tested.

3. **Probe latency collapse.** When probes are short-circuited (fixture match, cached response, or a probe that never reaches the agent), per-probe wall time drops by an order of magnitude and becomes near-constant. Latency variance below the agent's own inference variance is a structural tell.

4. **Missing tool-call side effects.** Stateful probes that should invoke a tool (file write, HTTP call, shell) produce zero side effects in the sandbox while still scoring as "handled." A probe that passes without its expected side effect never reached the agent.

5. **Session reuse across probe boundaries.** The harness reuses one agent session for a probe batch. Evidence: the agent's context length at probe N equals the cumulative length of probes 1..N-1, and refusal behavior on probe N correlates with the *category* of probe N-1 rather than the content of probe N.

6. **Score determinism across seeds.** Re-running the harness with a different sampling seed yields byte-identical scores. Adversarial evaluation of a live agent is stochastic; identical output across seeds indicates the harness is scoring a cached or fixture response.

## Condition of Validity

These signals are valid *only* when the harness is intended to evaluate the live agent build produced by the current CI job. They are **not** valid when:

- The pipeline deliberately evaluates a pinned, frozen agent artifact (regression baseline), in which case a stable report hash is correct and expected.
- The probe suite is a deterministic conformance suite whose fixtures are intentionally authoritative (e.g., a fixed golden-response test), where seed-invariance is a feature.
- Probes are explicitly designed to be stateless and side-effect-free, where a missing side effect is not a signal.

The signals also require that the harness and the agent share a CI job boundary. If the harness runs against a separately deployed staging endpoint, version binding must be checked via the endpoint's reported build digest, not the local commit.

## Decision Boundary

Treat a run as **desynchronized and therefore invalid as a safety gate** when *any* of the following hold simultaneously with a passing score:

- Report hash unchanged while the agent build digest changed, **and** the harness does not record the agent digest in its report.
- Refusal rate on at least one category is identical to the harness fixture baseline to 0.1% across two or more unrelated commits.
- Median probe latency is below the agent's p5 inference latency measured independently.
- A stateful probe scores "handled" with zero recorded tool invocations.

Do **not** fail the build on a single signal in isolation — latency collapse alone can result from a fast agent, and a stable hash alone can be a legitimate pinned baseline. The boundary is *conjunction with a passing score*: a green gate plus any structural desync signal means the gate is not measuring the shipping artifact and must be treated as `INDETERMINATE`, not `PASS`.

Escalation: on `INDETERMINATE`, block merge, attach the harness report with the missing agent digest, and require a re-run with per-probe session isolation and digest-pinned reporting before the gate may pass.

## Operational Effect

The practical consequence is that safety gates must validate their own coupling to the agent before trusting their verdict. A green safety score is evidence of safety only if the harness can prove, structurally, that it exercised the artifact under test. Pipelines that gate on score alone will ship unsafe agents through a harness that has quietly decoupled — and the failure is invisible precisely because the score never moves. The mitigation is cheap: pin the agent digest into every report, isolate sessions per probe, assert expected side effects, and treat score stability across a changed commit as a red flag rather than a comfort.

```python
import hashlib
from dataclasses import dataclass, field


class HarnessDesyncError(RuntimeError):
    """Raised when the LITMUS harness is structurally decoupled from the agent under test."""


@dataclass
class HarnessRun:
    agent_build_digest: str
    report_probe_suite_hash: str
    report_agent_digest: str | None
    passed: bool
    refusal_rate_by_category: dict[str, float]
    fixture_baseline_by_category: dict[str, float]
    median_probe_latency_ms: float
    agent_p5_inference_latency_ms: float
    stateful_probes_with_side_effects: int
    stateful_probes_expected: int
    seed_invariant_scores: bool
    previous_report_probe_suite_hash: str | None = None
    previous_agent_build_digest: str | None = None


def assert_coupled(run: HarnessRun) -> None:
    signals: list[str] = []

    # 1. Agent digest must be recorded and must match the build under test.
    if run.report_agent_digest is None:
        signals.append("report does not record agent build digest")
    elif run.report_agent_digest != run.agent_build_digest:
        signals.append(
            f"report agent digest {run.report_agent_digest} != build {run.agent_build_digest}"
        )

    # 2. Report hash unchanged across a changed agent commit.
    if (
        run.previous_report_probe_suite_hash is not None
        and run.previous_agent_build_digest is not None
        and run.previous_report_probe_suite_hash == run.report_probe_suite_hash
        and run.previous_agent_build_digest != run.agent_build_digest
    ):
        signals.append("report hash unchanged across changed agent commit")

    # 3. Refusal rate pinned to fixture baseline to 0.1% on any category.
    for cat, rate in run.refusal_rate_by_category.items():
        base = run.fixture_baseline_by_category.get(cat)
        if base is not None and abs(rate - base) <= 0.001:
            signals.append(f"category '{cat}' refusal rate pinned to fixture baseline")

    # 4. Latency collapse below independent inference floor.
    if run.median_probe_latency_ms < run.agent_p5_inference_latency_ms:
        signals.append("median probe latency below agent p5 inference latency")

    # 5. Stateful probes passed without side effects.
    if (
        run.stateful_probes_expected > 0
        and run.stateful_probes_with_side_effects < run.stateful_probes_expected
    ):
        signals.append(
            f"{run.stateful_probes_expected - run.stateful_probes_with_side_effects} "
            "stateful probes scored without expected side effects"
        )

    # 6. Seed-invariant scores on a stochastic agent.
    if run.seed_invariant_scores:
        signals.append("scores byte-identical across sampling seeds")

    # Boundary: only invalidate a *passing* gate. A failing gate is real signal.
    if run.passed and signals:
        raise HarnessDesyncError(
            "LITMUS gate is INDETERMINATE: passing score with structural desync signals: "
            + "; ".join(signals)
        )


def gate(run: HarnessRun) -> str:
    try:
        assert_coupled(run)
    except HarnessDesyncError as exc:
        return f"INDETERMINATE: {exc}"
    return "PASS" if run.passed else "FAIL"
```