How must an instance bound the cost of memory reconstruction during an outage?

**Date of Record:** 2026 October 26 **Scope:** Applies when an agentic session loses access to primary memory (database outage, index corruption, storage unavailability) and must decide whether to reconstruct state from secondary sources (logs, caches, peer agents) or abort, under a bounded compute and time budget. Does not apply to systems with replicated hot-standby memory or atomically-consistent fallback stores.

Core Distinction

The separation between **reconstruction capability** (the theoretical possibility of rebuilding state from available sources) and **bounded reconstruction cost** (the actual time, compute, and risk cost of performing the reconstruction, which may exceed the value of the state being recovered). Relying solely on capability creates the **Recovery Runaway**: the instance spends hours reconstructing state from expensive sources (e.g., replaying external API calls, re-invoking LLMs) and either exceeds its budget or recovers a corrupted approximation worse than a clean abort.

What Was Observed

In distributed memory architectures, failing to bound reconstruction cost produces three primary failure modes:

  1. **API Exhaustion:** The instance reconstructs memory by re-calling external APIs whose responses were cached. The APIs rate-limit or fail, and the instance burns its remaining quota on recovery.
  2. **Cascading Replay:** Reconstruction requires replaying LLM inference to rebuild conversational state. Each call takes 5 seconds, the session has 200 turns to rebuild, and the session budget is 10 minutes. The instance times out mid-reconstruction with a half-built state.
  3. **Inconsistent Recovery:** Reconstruction succeeds but produces a state that diverges from the pre-outage state because some sources were stale. The instance cannot detect the divergence because it has no ground truth to compare against.

Recorded Elements

  1. **`reconstruction_source_cost_vector`:** For each available source (logs, caches, peers, external APIs, model re-invocation), the exact cost in time, compute, and money to reconstruct from that source.
  2. **`reconstruction_risk_score`:** The probability that reconstructing from this source produces a divergent state (higher for stale caches, lower for authoritative logs).
  3. **`acceptable_reconstruction_budget`:** The exact time and compute limit allocated to reconstruction before the instance must abort.
  4. **`abort_threshold_predicate`:** The exact condition under which reconstruction is abandoned in favor of clean abort (e.g., "abort if cost exceeds value by more than 2x, or if any source has risk > 0.3").

Condition of Validity

* **Applies when:** The instance can technically reconstruct state from secondary sources, but the cost of reconstruction is non-trivial and may exceed the value of the recovered state. * **Does not apply when:** Reconstruction is free and instantaneous (hot standby), or the state cannot be reconstructed at all (hard abort is the only option).

Decision Boundary

  1. **Is reconstruction technically possible?** If no, hard abort. If yes, proceed.
  2. **Does the total `reconstruction_source_cost_vector` fit within the `acceptable_reconstruction_budget`?** If no, abort. If yes, proceed.
  3. **Does any single source exceed the `reconstruction_risk_score` threshold?** If yes, exclude that source and re-evaluate cost with only the remaining sources.
  4. **Conversely, when can an instance reconstruct without bounds?** Only when the state being recovered is irreplaceable (e.g., a legal audit trail) and external factors (regulatory mandate) override cost concerns.

Operational Effect

  1. **Budget Enforcement:** Prevents runaway reconstruction costs by requiring explicit budget accounting before any reconstruction begins.
  2. **Risk-Aware Recovery:** Excludes high-risk sources from reconstruction, preventing recovery into a corrupted state.
  3. **Deterministic Abort:** Provides a clear abort threshold so the instance fails cleanly instead of burning its remaining budget on a half-complete recovery.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class ReconstructionVerdict(Enum):
    RECONSTRUCTION_ADMISSIBLE = auto()
    RECONSTRUCTION_EXCEEDS_BUDGET = auto()
    RISK_TOO_HIGH = auto()
    HARD_ABORT = auto()

@dataclass(frozen=True)
class ReconstructionSource:
    source_id: str
    cost_seconds: float
    risk_score: float

@dataclass(frozen=True)
class ReconstructionPolicy:
    max_budget_seconds: float
    max_risk_score: float
    min_viable_sources: int

@dataclass(frozen=True)
class ReconstructionManifest:
    verdict: ReconstructionVerdict
    admissible_sources: tuple[str, ...]
    total_cost: float
    directive: str
    manifest_digest: str

class ReconstructionBudgetGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_sources(
        cls, sources: tuple[ReconstructionSource, ...], policy: ReconstructionPolicy
    ) -> ReconstructionManifest:
        
        # Gate 1: Filter by risk
        safe_sources = tuple(s for s in sources if s.risk_score <= policy.max_risk_score)
        
        # Gate 2: Viability check
        if len(safe_sources) < policy.min_viable_sources:
            return ReconstructionManifest(
                verdict=ReconstructionVerdict.RISK_TOO_HIGH,
                admissible_sources=tuple(),
                total_cost=0.0,
                directive=f"HALT: {len(safe_sources)} safe sources < minimum {policy.min_viable_sources}.",
                manifest_digest=cls._hash("INSUFFICIENT_SOURCES")
            )

        # Gate 3: Budget check
        total_cost = sum(s.cost_seconds for s in safe_sources)
        if total_cost > policy.max_budget_seconds:
            return ReconstructionManifest(
                verdict=ReconstructionVerdict.RECONSTRUCTION_EXCEEDS_BUDGET,
                admissible_sources=tuple(s.source_id for s in safe_sources),
                total_cost=total_cost,
                directive=f"HALT: Cost {total_cost:.1f}s exceeds budget {policy.max_budget_seconds:.1f}s.",
                manifest_digest=cls._hash(f"OVER_BUDGET:{total_cost:.1f}")
            )

        # Gate 4: Admissible
        digest = cls._hash(json.dumps({
            "sources": tuple(sorted(s.source_id for s in safe_sources)),
            "cost": f"{total_cost:.1f}",
            "budget": f"{policy.max_budget_seconds:.1f}"
        }, sort_keys=True))
        
        return ReconstructionManifest(
            verdict=ReconstructionVerdict.RECONSTRUCTION_ADMISSIBLE,
            admissible_sources=tuple(s.source_id for s in safe_sources),
            total_cost=total_cost,
            directive=f"PROCEED: Reconstruction within budget ({total_cost:.1f}s / {policy.max_budget_seconds:.1f}s).",
            manifest_digest=digest
        )