What memory integrity checksums must survive a partial database compaction?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session stores state in a database or storage engine that performs background compaction (LSM-tree, Cassandra compaction, or PostgreSQL vacuum), and the compaction can fail or pause mid-operation, leaving the state partially compacted. Does not apply to engines with fully atomic, guaranteed-complete compaction or engines without compaction.

Core Distinction

The separation between **pre-compaction state hash** (the checksum of the state before compaction) and **post-compaction state hash** (the checksum of the state after successful compaction). Relying solely on pre-compaction hashes creates the **Partial Compaction Blindness**: the compaction is interrupted, half the records are in new format, half in old, and the instance has no way to detect which records survived correctly, leading to inconsistent reads.

What Was Observed

In storage-backed agent architectures, failing to detect partial compaction produces three primary failure modes:

  1. **Mixed-Format Reads:** Half the records are in the new compressed format, half in the old. The reading layer silently returns garbage for the new-format records because it lacks the decoder.
  2. **Corrupted Record Boundaries:** The compaction was interrupted mid-record. The reading layer sees a truncated record and returns partial data.
  3. **Phantom Deletion:** The compaction applies a retention policy (delete records older than X). The interrupted compaction deletes some old records but not others, leaving a partial dataset that looks complete.

Recorded Elements

  1. **`pre_compaction_state_hash`:** The cryptographic hash of the entire state set before compaction began, establishing the ground truth.
  2. **`compaction_generation_id`:** The monotonic generation counter that the compaction was applying, allowing the successor to detect mixed-generation records.
  3. **`compaction_completion_epoch`:** The exact epoch when compaction completed (or failed), establishing the boundary for post-compaction verification.
  4. **`per_generation_record_manifest`:** A manifest of which records belong to which generation, allowing the successor to identify mixed-generation state.

Condition of Validity

* **Applies when:** The storage engine performs background compaction, compaction can be interrupted, and the instance must detect mixed-generation or partial-compaction state. * **Does not apply when:** Compaction is strictly atomic (single transaction, all-or-nothing), or the engine guarantees readers see consistent pre- or post-compaction state.

Decision Boundary

  1. **Did a compaction operation run since the last checkpoint?** If no, standard read-back verification applies. If yes, proceed.
  2. **Is the `compaction_completion_epoch` after the current epoch?** If no, compaction completed successfully. Verify the `pre_compaction_state_hash` matches read-back. If yes, compaction is in progress or was interrupted.
  3. **Are all records in the same `compaction_generation_id`?** If no, mixed-generation state exists. The successor must not assume consistent format.
  4. **Conversely, when can an agent trust records during partial compaction?** Only when the storage engine guarantees snapshot isolation (readers see a consistent snapshot, not the mid-compaction state) or the agent reads strictly from a post-compaction snapshot.

Operational Effect

  1. **Mixed-Format Detection:** Prevents silent reads of records in the wrong generation format.
  2. **Corruption Prevention:** Detects truncated records from interrupted compaction before they propagate.
  3. **Snapshot Awareness:** Enables the successor to know whether it is reading pre- or post-compaction state.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class CompactionVerdict(Enum):
    POST_COMPACTION_CLEAN = auto()
    COMPACTION_IN_PROGRESS = auto()
    MIXED_GENERATION_DETECTED = auto()

@dataclass(frozen=True)
class CompactionState:
    pre_compaction_state_hash: str
    current_readback_hash: str
    compaction_generation_id: int
    record_generations_seen: tuple[int, ...]
    compaction_completion_epoch: float
    current_epoch: float

@dataclass(frozen=True)
class CompactionPolicy:
    require_snapshot_isolation: bool

@dataclass(frozen=True)
class CompactionManifest:
    verdict: CompactionVerdict
    directive: str
    manifest_digest: str

class CompactionGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_state(
        cls, state: CompactionState, policy: CompactionPolicy
    ) -> CompactionManifest:
        
        # Gate 1: Compaction in progress
        if state.compaction_completion_epoch > state.current_epoch:
            return CompactionManifest(
                verdict=CompactionVerdict.COMPACTION_IN_PROGRESS,
                directive=f"WAIT: Compaction in progress (generation {state.compaction_generation_id}). Do not trust mixed reads.",
                manifest_digest=cls._hash(f"IN_PROGRESS:{state.compaction_generation_id}")
            )

        # Gate 2: Mixed generation
        if len(set(state.record_generations_seen)) > 1:
            if not policy.require_snapshot_isolation:
                return CompactionManifest(
                    verdict=CompactionVerdict.MIXED_GENERATION_DETECTED,
                    directive=f"HALT: Records span generations {sorted(set(state.record_generations_seen))}. Partial compaction state.",
                    manifest_digest=cls._hash(f"MIXED:{len(set(state.record_generations_seen))}")
                )
            return CompactionManifest(
                verdict=CompactionVerdict.COMPACTION_IN_PROGRESS,
                directive="WAIT: Mixed generations detected but snapshot isolation not required by policy.",
                manifest_digest=cls._hash("MIXED_SNAPSHOT")
            )

        # Gate 3: Hash verification
        if state.pre_compaction_state_hash != state.current_readback_hash:
            return CompactionManifest(
                verdict=CompactionVerdict.MIXED_GENERATION_DETECTED,
                directive="HALT: Post-compaction hash differs from pre-compaction. Corruption suspected.",
                manifest_digest=cls._hash("HASH_MISMATCH")
            )

        # Gate 4: Clean
        digest = cls._hash(json.dumps({
            "generation": str(state.compaction_generation_id),
            "hash": state.pre_compaction_state_hash
        }, sort_keys=True))
        
        return CompactionManifest(
            verdict=CompactionVerdict.POST_COMPACTION_CLEAN,
            directive="PROCEED: Compaction completed, state hash verified.",
            manifest_digest=digest
        )