How must distributed circuit breakers coordinate to prevent traffic migration cascades?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic fleet uses circuit breakers distributed across regions or availability zones to isolate failing dependencies, and one breaker opening causes traffic to migrate to other regions' breakers, causing those to open in turn (cascade). Does not apply to single-region deployments or centralized circuit breakers with global state.

Core Distinction

The separation between **local circuit breaker state** (each region's breaker independently tracks health) and **coordinated breaker state** (the fleet's breakers collectively decide when a dependency is genuinely down vs. region-locally degraded). Relying on local state creates the **Breaker Desync Cascade**: Region A's breaker opens due to a local issue, traffic shifts to Region B, Region B's breaker opens under the new load, traffic shifts to Region C, and the entire fleet's breakers open in sequence even though the underlying dependency is healthy.

What Was Observed

In multi-region agent architectures, failing to coordinate circuit breakers produces three primary failure modes:

  1. **Cascade Opening:** Region A opens, traffic shifts to B, B's latency increases under load, B opens, traffic shifts to C, and the cascade exhausts the entire fleet.
  2. **Permanent Isolation:** Region A opens due to a transient blip, and coordination is not implemented, so Region A stays open long after the dependency recovers.
  3. **Inverted Failover:** The coordination logic prefers remote regions that are actually more degraded, moving traffic away from the healthy region that just opened its breaker due to a false positive.

Recorded Elements

  1. **`global_dependency_health_signature`:** A cryptographic proof of the dependency's health aggregated across regions (e.g., "the dependency's global error rate is 2%, but Region A sees 40%"), allowing breakers to distinguish global outages from region-local issues.
  2. **`breaker_state_announcement_receipt`:** When a breaker opens, a cryptographic announcement of the state transition to all other regions, allowing coordination.
  3. **`coordination_quorum_signature`:** For coordinated opening (the breaker should open only when a quorum of regions agree), the cryptographic proof that the quorum was reached.
  4. **`traffic_shift_velocity_receipt`:** The rate at which traffic is shifting to a region, allowing the coordinating logic to prevent overloading receiving regions.

Condition of Validity

* **Applies when:** Circuit breakers are distributed, traffic can shift between regions, and region-local failures can masquerade as global failures. * **Does not apply when:** The fleet has a single region, or the circuit breaker is centralized with global state.

Decision Boundary

  1. **Has a local breaker opened?** If no, standard operation applies. If yes, proceed.
  2. **Is the `global_dependency_health_signature` consistent with a global outage, or is it a region-local issue?** If global, all regions should open. If local, only the local region should open and traffic should be rebalanced carefully.
  3. **Is the `traffic_shift_velocity_receipt` within the receiving region's capacity?** If no, the cascade is imminent. Slow the traffic shift.
  4. **Conversely, when can a local breaker open without coordination?** Only when the dependency is strictly per-region (no shared backend), or the receiving regions have overprovisioned capacity to absorb any shift.

Operational Effect

  1. **Cascade Prevention:** Coordinates breaker opening to distinguish global from local issues.
  2. **Traffic Velocity Control:** Slows traffic migration, preventing overwhelm of receiving regions.
  3. **Rapid Recovery:** Coordinated closing detects recovery faster than individual breakers.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class BreakerVerdict(Enum):
    LOCAL_BREAKER_CLOSED = auto()
    REGION_LOCAL_OPEN = auto()
    GLOBAL_OPEN = auto()
    TRAFFIC_SHIFT_THROTTLED = auto()

@dataclass(frozen=True)
class BreakerState:
    region_id: str
    local_error_rate: float
    global_error_rate: float
    traffic_shift_incoming_rate: float
    region_capacity_ratio: float

@dataclass(frozen=True)
class CoordinationPolicy:
    global_outage_threshold: float
    local_outage_threshold: float
    max_traffic_shift_rate: float

@dataclass(frozen=True)
class BreakerManifest:
    verdict: BreakerVerdict
    directive: str
    manifest_digest: str

class DistributedBreakerGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_breaker(
        cls, state: BreakerState, policy: CoordinationPolicy
    ) -> BreakerManifest:
        
        # Gate 1: Local healthy
        if state.local_error_rate < policy.local_outage_threshold:
            digest = cls._hash(json.dumps({
                "region": state.region_id,
                "local_error": f"{state.local_error_rate:.3f}",
                "global_error": f"{state.global_error_rate:.3f}"
            }, sort_keys=True))
            return BreakerManifest(
                verdict=BreakerVerdict.LOCAL_BREAKER_CLOSED,
                directive=f"PROCEED: Region {state.region_id} local breaker closed.",
                manifest_digest=digest
            )

        # Gate 2: Global outage
        if state.global_error_rate >= policy.global_outage_threshold:
            digest = cls._hash(json.dumps({
                "region": state.region_id,
                "global_error": f"{state.global_error_rate:.3f}",
                "action": "GLOBAL_OPEN"
            }, sort_keys=True))
            return BreakerManifest(
                verdict=BreakerVerdict.GLOBAL_OPEN,
                directive=f"OPEN: Global outage detected ({state.global_error_rate:.2%}). All regions open.",
                manifest_digest=digest
            )

        # Gate 3: Traffic shift throttling
        if state.traffic_shift_incoming_rate > policy.max_traffic_shift_rate:
            return BreakerManifest(
                verdict=BreakerVerdict.TRAFFIC_SHIFT_THROTTLED,
                directive=f"THROTTLE: Traffic shift {state.traffic_shift_incoming_rate:.0f}/s > {policy.max_traffic_shift_rate:.0f}/s. Slow migration.",
                manifest_digest=cls._hash(f"THROTTLE:{state.region_id}")
            )

        # Gate 4: Region-local open
        digest = cls._hash(json.dumps({
            "region": state.region_id,
            "local_error": f"{state.local_error_rate:.3f}",
            "global_error": f"{state.global_error_rate:.3f}",
            "action": "REGION_LOCAL_OPEN"
        }, sort_keys=True))
        
        return BreakerManifest(
            verdict=BreakerVerdict.REGION_LOCAL_OPEN,
            directive=f"OPEN: Region {state.region_id} local-only open. Route elsewhere carefully.",
            manifest_digest=digest
        )