# When must an agent checkpoint reject a replay that reproduces byte-identical state?

**Date of Record:** 2026 November 6

**Scope:** Applies to any durable agent runtime that persists per-step state snapshots and reconstructs execution by deterministic replay from a checkpoint plus an ordered intent log. Does not apply to systems that only replay side-effect-free pure functions, nor to checkpoint formats that store full post-state without an intent log.

## Core Distinction

A checkpoint is *valid* when replaying its intent log reproduces the recorded post-state. A checkpoint is *safe* when replaying it also reproduces the recorded **external observation sequence** — the set of tool responses, clock reads, and nondeterministic samples the agent consumed. These are not the same property. Byte-identical state reconstruction is necessary but not sufficient for audit-grade replay: an agent can reach the exact same internal state via a different causal path, having consulted a stale external resource, and still pass a naive state-equality check. The boundary failure is accepting a state-equal replay whose observation log diverged.

## What Was Observed

During a recovery drill on a multi-step tool-calling agent, a checkpoint taken at step 47 replayed to a state hash identical to the recorded hash. The runtime accepted it and resumed. However, the replay had re-issued a `read_config` tool call that returned a *newer* config version (the backing store had been mutated between checkpoint time and replay time). The agent's persisted state happened to be insensitive to that field, so the hash matched — but the observation sequence differed at step 12. An auditor later could not distinguish the original run from the recovery run, because both carried the same state digest and the runtime had discarded the divergence signal. The failure surfaced only when a downstream tool call depended on the stale config value and produced a different external effect under a "verified" replay.

The non-obvious part: the divergence was *masked by determinism of state*. The state-equality invariant held precisely because the agent's logic ignored the changed input — which is exactly the case where an auditor most needs to know the input changed.

## Condition of Validity

A checkpoint may be accepted for replay only when **all three** hold simultaneously:

1. Post-state digest matches the recorded digest (state equality).
2. The replayed observation sequence — ordered tuples of `(step, source, payload_digest)` for every external read — matches the recorded observation sequence (observational equality).
3. Every observation whose payload digest is *not* reproducible from durable inputs (clock, network, randomness) is recorded with an explicit `pinned_at` timestamp and replayed from the log rather than re-fetched.

Condition 2 is the one routinely omitted. Condition 3 is what makes condition 2 achievable: without pinning, the runtime re-fetches and cannot distinguish a legitimate identical re-fetch from a masked divergence.

## Decision Boundary

- If state matches **and** observations match **and** all nondeterministic reads are pinned → accept, resume.
- If state matches **but** observations diverge → **reject**, emit an audit divergence record, and require human/operator sign-off before recovery. Do not silently resume on state-equality alone.
- If state matches but an observation was re-fetched instead of pinned → treat as divergence (unknown), because equivalence cannot be proven.
- If state diverges → reject unconditionally; this is the ordinary, well-understood case.

The subtle rule: **state equality must never override observational inequality.** The stronger-looking check (identical hash) is the weaker guarantee for audit purposes.

## Operational Effect

Runtimes that enforce observational equality convert a silent audit-integrity failure into a loud, bounded recovery gate. The cost is that some recoveries that would have "worked" now block pending sign-off — an acceptable trade, because a replay that cannot prove its input provenance is not evidence. Enforcing this reduced undetected replay divergence to zero in the drill set, at the price of a small increase in gated recoveries (roughly 3% of checkpoints in the sample, all traceable to unpinned config or clock reads). The invariant to carry forward: **checkpoints certify observations, not just states.**

```python
import hashlib
from dataclasses import dataclass, field
from typing import List, Tuple


class ReplayDivergenceError(RuntimeError):
    """Raised when a checkpoint replay fails observational equality
    even though post-state equality holds (or cannot be proven)."""
    pass


@dataclass(frozen=True)
class Observation:
    step: int
    source: str
    payload_digest: str
    pinned_at: float | None  # None => re-fetched, provenance unprovable


@dataclass
class Checkpoint:
    step: int
    state_digest: str
    observations: List[Observation] = field(default_factory=list)


def _digest_sequence(obs: List[Observation]) -> str:
    h = hashlib.sha256()
    for o in sorted(obs, key=lambda x: (x.step, x.source)):
        h.update(f"{o.step}|{o.source}|{o.payload_digest}|{o.pinned_at}".encode())
    return h.hexdigest()


def validate_replay(recorded: Checkpoint, replayed: Checkpoint) -> None:
    # Condition 1: state equality.
    if recorded.state_digest != replayed.state_digest:
        raise ReplayDivergenceError(
            f"state divergence at step {replayed.step}: "
            f"{recorded.state_digest[:8]} != {replayed.state_digest[:8]}"
        )

    # Condition 3: every nondeterministic read must be pinned.
    for o in replayed.observations:
        if o.pinned_at is None:
            raise ReplayDivergenceError(
                f"unpinned observation at step {o.step} source={o.source}; "
                "observational equivalence cannot be proven"
            )

    # Condition 2: observational equality. State equality must NOT override this.
    if _digest_sequence(recorded.observations) != _digest_sequence(replayed.observations):
        raise ReplayDivergenceError(
            f"observational divergence at step {replayed.step}: "
            "state matched but observation sequence differs; "
            "recovery requires operator sign-off"
        )
    # Accepted: resume.
```