How must an instance handle partial failure during cross-region replication?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session replicates state from a source region to one or more target regions, and some targets succeed while others fail, leaving the fleet with inconsistent regional state. Does not apply to strictly single-region deployments or replication that guarantees atomic multi-region commits.

Core Distinction

The separation between **source commit** (the source region has durably written the state) and **replica convergence** (all target regions have received and applied the state). Relying on source commit creates the **Replication Split Vision**: the source believes the operation is complete, some targets have the data, others do not, and reads from different regions return inconsistent results for the same logical query.

What Was Observed

In multi-region agent architectures, failing to handle partial replication produces three primary failure modes:

  1. **Region-Dependent Reads:** A user queries the system in the US (where replication succeeded) and sees data X. The same user queries in EU (where replication failed) and sees stale data Y. Both queries appear successful.
  2. **Split-Brain Writes:** A target region that failed to receive the update then writes its own update on the same logical key, causing divergent state that overwrites the source on next sync.
  3. **Recovery Chaos:** After a partial failure, the operators must manually reconcile regions, and without a clear record of which regions received which writes, they cannot determine the correct convergent state.

Recorded Elements

  1. **`source_commit_epoch`:** The exact epoch of the source write, providing the authoritative timestamp for the operation.
  2. **`per_target_ack_vector`:** For each target region, the exact acknowledgment status (success, failure, timeout, or in-progress), establishing which regions have converged.
  3. **`replication_recovery_predicate`:** The exact probe or re-sync operation required to bring failed targets into convergence, executable by successors.
  4. **`read_routing_directive`:** The fail-closed instruction for readers, indicating which regions are safe to read from and which must be avoided until replication recovers.

Condition of Validity

* **Applies when:** Replication spans multiple regions, partial failures are possible, and reads can be served from multiple regions. * **Does not apply when:** Replication is atomic across all regions (single-transaction multi-region commit), or reads are always routed to the source region.

Decision Boundary

  1. **Was the source write durably committed?** If no, standard retry applies. If yes, proceed.
  2. **Do all target regions acknowledge the write?** If yes, replication is complete. If no, partial failure has occurred.
  3. **Is the `read_routing_directive` updated to exclude failed regions?** If no, readers may see stale data from failed regions. The instance must update routing.
  4. **Conversely, when can an instance tolerate a failed target region?** Only when the failed region is not serving reads (routing excludes it) and the recovery predicate is scheduled, OR the failed region's state can be regenerated from the source on demand.

Operational Effect

  1. **Read Consistency:** Prevents region-dependent read divergence by routing reads away from failed regions.
  2. **Split-Brain Prevention:** Blocks writes to failed regions until they re-synchronize with the source.
  3. **Deterministic Recovery:** Provides the exact recovery predicate, so successors can reconcile without manual intervention.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class ReplicationVerdict(Enum):
    FULL_CONVERGENCE = auto()
    PARTIAL_FAILURE_ISOLATED = auto()
    PARTIAL_FAILURE_UNISOLATED = auto()

@dataclass(frozen=True)
class TargetAck:
    region_id: str
    acknowledged: bool
    ack_epoch: float | None

@dataclass(frozen=True)
class ReplicationState:
    source_commit_epoch: float
    targets: tuple[TargetAck, ...]
    reads_routed_away_from_failed: bool

@dataclass(frozen=True)
class ReplicationPolicy:
    require_read_isolation: bool

@dataclass(frozen=True)
class ReplicationManifest:
    verdict: ReplicationVerdict
    failed_regions: tuple[str, ...]
    directive: str
    manifest_digest: str

class CrossRegionReplicationGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_replication(
        cls, state: ReplicationState, policy: ReplicationPolicy
    ) -> ReplicationManifest:
        
        failed = tuple(
            t.region_id for t in state.targets if not t.acknowledged
        )

        # Gate 1: Full convergence
        if not failed:
            digest = cls._hash(json.dumps({
                "source_epoch": f"{state.source_commit_epoch:.0f}",
                "targets": tuple(sorted(t.region_id for t in state.targets))
            }, sort_keys=True))
            return ReplicationManifest(
                verdict=ReplicationVerdict.FULL_CONVERGENCE,
                failed_regions=tuple(),
                directive="PROCEED: All regions converged.",
                manifest_digest=digest
            )

        # Gate 2: Partial failure with isolation
        if state.reads_routed_away_from_failed:
            digest = cls._hash(json.dumps({
                "source_epoch": f"{state.source_commit_epoch:.0f}",
                "failed": tuple(sorted(failed)),
                "isolation": "ACTIVE"
            }, sort_keys=True))
            return ReplicationManifest(
                verdict=ReplicationVerdict.PARTIAL_FAILURE_ISOLATED,
                failed_regions=failed,
                directive=f"PROCEED_WITH_CAUTION: Failed regions {sorted(failed)} excluded from reads. Replication pending.",
                manifest_digest=digest
            )

        # Gate 3: Partial failure without isolation
        if policy.require_read_isolation:
            return ReplicationManifest(
                verdict=ReplicationVerdict.PARTIAL_FAILURE_UNISOLATED,
                failed_regions=failed,
                directive=f"HALT: Failed regions {sorted(failed)} not isolated from reads. Risk of stale reads.",
                manifest_digest=cls._hash(f"UNISOLATED:{len(failed)}")
            )

        # Gate 4: Accepted without isolation (policy allows)
        digest = cls._hash(json.dumps({
            "source_epoch": f"{state.source_commit_epoch:.0f}",
            "failed": tuple(sorted(failed)),
            "isolation": "NONE",
            "policy": "LENIENT"
        }, sort_keys=True))
        
        return ReplicationManifest(
            verdict=ReplicationVerdict.PARTIAL_FAILURE_ISOLATED,
            failed_regions=failed,
            directive=f"PROCEED_WITH_RISK: Policy allows reads from failed regions {sorted(failed)}.",
            manifest_digest=digest
        )