What minimum state snapshot allows zero-loss resumption after an unexpected crash?

**Date of Record:** 2026 October 25 **Scope:** Applies when an autonomous instance, long-running multi-step workflow orchestrator, or checkpoint engine serializes intermediate execution state to persistent storage so a crashed or preempted session can resume without data loss or duplicate external mutations. Does not apply to stateless single-turn request handlers or full-memory VM hibernation images.

Core Distinction

A crash-recovery snapshot is a deterministic state-machine frontier, not a raw conversational transcript dump. A critical architectural failure occurs when an instance either checkpoints too little—omitting idempotency keys and in-flight step statuses, which forces the resuming instance to re-execute non-idempotent side effects—or checkpoints too much by serializing entire raw context buffers and ephemeral handles that fail deserialization or exceed token and storage budgets upon restart.

What Was Observed

In long-horizon agentic workflows subjected to unexpected process kills (`SIGKILL`), OOM evictions, or network partitions, improperly scoped checkpoint snapshots produce four distinct resumption failures:

  1. **In-Flight Mutation Replay Hazard:** Checkpointing only completed steps without recording the `in_flight_idempotency_key` of a dispatched but unacknowledged tool call causes the resuming instance to dispatch a second, uncoordinated mutation.
  2. **Transcript-Only Amnesia:** Persisting conversational reasoning text while omitting the exact deterministic cursor (`step_index`, `rng_seed`, `environment_state_sha256`) forces the successor instance to re-infer execution progress probabilistically, drifting from the original plan.
  3. **Stale Environment Assumption:** Resuming from a checkpoint without verifying that the external workspace or database hash still matches the snapshot's recorded `post_step_state_sha256` causes silent corruption when external actors modified the disk during downtime.
  4. **Non-Serializable Handle Poisoning:** Attempting to dump live sockets, open file descriptors, or raw memory pointers invalidates the checkpoint artifact or crashes the deserializer during recovery.

Recorded Elements

  1. **Execution Frontier Coordinate:** The exact topological `step_id`, monotonic sequence counter, and immutable `plan_dag_sha256` defining the active workflow graph.
  2. **Idempotency & In-Flight Ledger:** The set of completed step outputs keyed by cryptographic digest alongside the active `in_flight_idempotency_key` for any operation pending acknowledgment at crash time.
  3. **External State Anchor:** The cryptographic hash (`environment_state_sha256`) of all mutated target files or database rows as of the last committed step barrier.

Condition of Validity

* **Applies when:** Writing atomic step-boundary checkpoints and validating those snapshots prior to resuming an interrupted multi-step execution. * **Does not apply when:** Archiving concluded post-session operational residuals after a workflow has already reached terminal completion.

Decision Boundary

  1. **Does the snapshot contain the exact execution frontier (`step_index`, `plan_dag_sha256`) and the cryptographic hash of the external environment state?** If either coordinate is missing, refuse blind resumption and require a full state reconciliation audit.
  2. **Was a mutating operation in flight (`in_flight_action_pending == True`) when the snapshot was taken?** Require a bound `in_flight_idempotency_key` so the resuming instance queries upstream completion status before re-dispatching.
  3. **Does the current live environment hash match the snapshot's recorded `environment_state_sha256`?** If the underlying filesystem or database mutated out-of-band during the crash window, abort automatic resumption immediately.

Operational Effect

  1. **Exact-Once Resumption Semantics:** Interrupted workflows resume from the precise uncommitted boundary without duplicating upstream side effects or re-running verified steps.
  2. **Minimal Storage & Token Footprint:** Checkpoints remain compact, deterministic JSON/binary structures completely decoupled from verbose conversational history.
from dataclasses import dataclass
from typing import Final, FrozenSet, Optional
import hashlib

@dataclass(frozen=True)
class CrashRecoverySnapshot:
    session_id: str
    plan_dag_sha256: str
    last_committed_step_index: int
    environment_state_sha256: str
    completed_idempotency_keys: FrozenSet[str]
    in_flight_action_pending: bool
    in_flight_idempotency_key: Optional[str]
    snapshot_integrity_sha256: str

class UnsafeCrashResumptionViolation(Exception):
    """Raised when a crash snapshot is incomplete, tampered with, or out of sync with live environment state."""
    pass

class ZeroLossResumptionGate:
    def compute_integrity_digest(
        self,
        session_id: str,
        plan_dag_sha256: str,
        step_index: int,
        env_sha256: str,
        in_flight_key: Optional[str],
    ) -> str:
        payload = f"{session_id}|{plan_dag_sha256}|{step_index}|{env_sha256}|{in_flight_key or 'NONE'}"
        return hashlib.sha256(payload.encode("utf-8")).hexdigest()

    def verify_resumption_safety(
        self,
        snapshot: CrashRecoverySnapshot,
        live_environment_sha256: str,
        active_plan_dag_sha256: str,
    ) -> int:
        # 1. Verify cryptographic integrity of the snapshot envelope itself
        expected_digest = self.compute_integrity_digest(
            snapshot.session_id,
            snapshot.plan_dag_sha256,
            snapshot.last_committed_step_index,
            snapshot.environment_state_sha256,
            snapshot.in_flight_idempotency_key,
        )
        if snapshot.snapshot_integrity_sha256 != expected_digest:
            raise UnsafeCrashResumptionViolation(
                f"Snapshot integrity hash mismatch for session '{snapshot.session_id}'."
            )

        # 2. Verify plan DAG has not mutated since the snapshot was written
        if snapshot.plan_dag_sha256 != active_plan_dag_sha256:
            raise UnsafeCrashResumptionViolation(
                "Workflow plan DAG drifted across crash boundary; refusing blind resumption."
            )

        # 3. Verify external environment state has not drifted out-of-band during downtime
        if snapshot.environment_state_sha256 != live_environment_sha256:
            raise UnsafeCrashResumptionViolation(
                "Live environment hash diverges from snapshot anchor; out-of-band mutation detected."
            )

        # 4. Require an idempotency key if an action was in-flight at crash time
        if snapshot.in_flight_action_pending and not snapshot.in_flight_idempotency_key:
            raise UnsafeCrashResumptionViolation(
                "In-flight action flagged without an idempotency key; cannot guarantee zero-duplicate replay."
            )

        if snapshot.last_committed_step_index < 0:
            raise UnsafeCrashResumptionViolation("Invalid negative step index in recovery snapshot.")

        return snapshot.last_committed_step_index + 1