What memory invariants must survive a silent storage backend upgrade?

**Date of Record:** 2026 October 26 **Scope:** Applies when an agentic session persists state to a storage backend (database, object store, vector DB) that is silently upgraded by the provider (engine version, storage format, compression algorithm), and the successor must detect the upgrade and reconcile any state that was affected. Does not apply to strictly immutable storage backends or environments where storage upgrades are coordinated with the session.

Core Distinction

The separation between **storage availability** (the backend is reachable and returns data) and **storage format continuity** (the data returned is in the exact format the instance wrote, with the same serialization, compression, and encoding). Relying solely on availability creates the **Silent Format Drift**: the backend is upgraded, the data is technically readable, but subtle changes (float precision, date encoding, unicode normalization) silently corrupt the state. The successor inherits data that looks correct but has been altered by the storage layer.

What Was Observed

In cloud-native memory architectures, failing to detect silent storage backend upgrades produces three primary failure modes:

  1. **Float Precision Loss:** The storage backend upgrades to a new column type that truncates floats from 64-bit to 32-bit. Recorded scores, timestamps, and monetary values lose precision silently.
  2. **Encoding Change:** The backend upgrades from UTF-8 to a compressed encoding that alters unicode normalization. Stored strings that were NFC become NFD, breaking exact-match queries.
  3. **Compression Algorithm Change:** The backend upgrades its compression from gzip to zstd. Decompressed payloads are byte-identical but the underlying storage format on disk has changed, breaking tools that inspect the raw bytes.

Recorded Elements

  1. **`backend_version_at_write`:** The exact backend engine version and storage format at write time, allowing the successor to compare against the current version.
  2. **`serialization_round_trip_hash`:** The hash of the payload after a full write-read round-trip at write time, establishing the ground truth for detecting format drift.
  3. **`normalization_form_signature`:** The exact Unicode normalization form and any other encoding guarantees the storage backend promised at write time.
  4. **`format_drift_detector_predicate`:** The exact read-back verification the successor must perform on a sample of records to detect silent format changes.

Condition of Validity

* **Applies when:** The storage backend is provider-managed (can be upgraded silently), the data is precision-sensitive (numerics, timestamps, strings), and the successor must guarantee the data survived the upgrade intact. * **Does not apply when:** The storage backend is provider-locked to a version the agent controls, or the data is strictly safe against format drift (e.g., cryptographic hashes of blobs with no formatting).

Decision Boundary

  1. **Has the storage backend version changed since `backend_version_at_write`?** If no, standard reads apply. If yes, proceed.
  2. **Does a sample read-back match the `serialization_round_trip_hash`?** If yes, the format survived. If no, silent format drift has occurred.
  3. **Does the sample match the `normalization_form_signature`?** If no, normalization has drifted. The successor must apply re-normalization before use.
  4. **Conversely, when can a successor trust records after a silent upgrade without read-back verification?** Only when the backend provides cryptographic proof of format continuity (e.g., signed checksums of the stored bytes), or the records are explicitly marked as non-precision-sensitive.

Operational Effect

  1. **Silent Corruption Prevention:** Detects format drift after upgrades, preventing the successor from propagating corrupted state.
  2. **Normalization Enforcement:** Ensures unicode normalization survives the upgrade, preserving exact-match query correctness.
  3. **Precision Preservation:** Detects numeric precision loss, preserving the fidelity of recorded scores, timestamps, and monetary values.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class FormatDriftVerdict(Enum):
    FORMAT_STABLE = auto()
    SILENT_FORMAT_DRIFT = auto()
    NORMALIZATION_DRIFT = auto()
    BACKEND_VERSION_CHANGED = auto()

@dataclass(frozen=True)
class StorageWriteRecord:
    record_id: str
    backend_version_at_write: str
    serialization_round_trip_hash: str
    normalization_form_signature: str

@dataclass(frozen=True)
class CurrentBackendState:
    current_backend_version: str
    sample_readback_hash: str
    current_normalization_form: str

@dataclass(frozen=True)
class FormatDriftManifest:
    verdict: FormatDriftVerdict
    directive: str
    manifest_digest: str

class SilentStorageUpgradeGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_record(
        cls, record: StorageWriteRecord, state: CurrentBackendState
    ) -> FormatDriftManifest:
        
        # Gate 1: Backend version change
        if record.backend_version_at_write != state.current_backend_version:
            # Version changed, require readback verification
            if record.serialization_round_trip_hash != state.sample_readback_hash:
                return FormatDriftManifest(
                    verdict=FormatDriftVerdict.SILENT_FORMAT_DRIFT,
                    directive=f"HALT: Backend version {record.backend_version_at_write} -> {state.current_backend_version}. Readback hash mismatch. Silent drift.",
                    manifest_digest=cls._hash(f"DRIFT:{record.record_id}")
                )
            if record.normalization_form_signature != state.current_normalization_form:
                return FormatDriftManifest(
                    verdict=FormatDriftVerdict.NORMALIZATION_DRIFT,
                    directive=f"HALT: Normalization {record.normalization_form_signature} -> {state.current_normalization_form}.",
                    manifest_digest=cls._hash(f"NORM_DRIFT:{record.record_id}")
                )
            return FormatDriftManifest(
                verdict=FormatDriftVerdict.BACKEND_VERSION_CHANGED,
                directive=f"PROCEED: Backend upgraded to {state.current_backend_version} but format verified stable.",
                manifest_digest=cls._hash(f"UPGRADED_SAFE:{record.record_id}")
            )

        # Gate 2: No version change, standard verification
        digest = cls._hash(json.dumps({
            "record": record.record_id,
            "version": record.backend_version_at_write
        }, sort_keys=True))
        
        return FormatDriftManifest(
            verdict=FormatDriftVerdict.FORMAT_STABLE,
            directive="PROCEED: Backend version unchanged. Format stable.",
            manifest_digest=digest
        )