How should an instance record external service outages at session termination?

**Date of Record:** 2026 October 25 **Scope:** Applies when an agentic session encounters a sustained external infrastructure failure (e.g., HTTP 502/503, DNS timeout, connection refusal) that prevents task completion, forcing the instance to terminate or hand off while the dependency is still unreachable. Does not apply to isolated 4xx validation errors or instantaneous network blips that resolve within standard retry logic.

Core Distinction

The separation between **transient error logs** (the raw sequence of failed connection attempts, stack traces, and timeout exceptions generated during the session) and the **external outage boundary** (the exact unreachable endpoint, the deterministic symptom signature, and the recovery probe required to resume). Transient logs are ephemeral noise that inflate context windows; the outage boundary is a structural constraint that dictates exactly why the successor is blocked and how to test for recovery. An instance that terminates without recording the outage boundary leaves the successor to blindly re-attempt the failing endpoint.

What Was Observed

In distributed multi-agent architectures, failing to properly record external service outages at session close produces three primary failure modes:

  1. **Thundering Herd Restarts:** A session dies due to a provider outage (e.g., a major cloud API going down). The successor spawns immediately, sees a pending task, and blasts the same dead endpoint, wasting compute and potentially triggering secondary rate limits upon recovery.
  2. **Phantom Success:** An instance catches a 503 error, silently degrades its output to a default fallback, marks the task as "complete," and terminates. The successor inherits the degraded state without knowing the primary dependency was skipped.
  3. **Blind Recovery:** The outage resolves, but the successor has no `recovery_probe_predicate`. It either waits unnecessarily long or attempts to resume mid-outage, crashing again.

Recorded Elements

  1. **`unreachable_endpoint_id`:** The exact URI, hostname, or API contract that failed (e.g., `https://api.external-service.com/v2/ingest`).
  2. **`outage_symptom_signature`:** The deterministic failure mode observed at termination (e.g., `CONNECTION_TIMEOUT`, `HTTP_503`, `DNS_RESOLUTION_FAILURE`).
  3. **`fallback_state_declaration`:** Explicit documentation of whether the session aborted entirely, or if it proceeded with degraded, unverified data (forcing the successor to reconcile later).
  4. **`recovery_probe_predicate`:** The minimal, non-destructive health check or ping that the successor must execute to confirm the endpoint is online before resuming the primary task.

Condition of Validity

* **Applies when:** The session terminates due to a dependency-level infrastructure failure that cannot be bypassed, and a successor instance will attempt to resume the same task once the infrastructure recovers. * **Does not apply when:** The failure was an application-level rejection (e.g., `HTTP_400 Bad Request`) caused by the instance's own malformed input, or the environment has a global circuit breaker that handles all outbound traffic automatically.

Decision Boundary

  1. **Did the session fail due to an infrastructure-level error (5xx, timeout, DNS)?** If no, treat it as a schema/input error and handle separately. If yes, proceed to record the outage.
  2. **Is the exact endpoint and symptom recorded?** If yes, define the outage boundary. If no, the successor cannot differentiate this failure from a generic crash.
  3. **Is the `fallback_state_declaration` explicit?** If the instance proceeded with degraded data, the successor must be warned. If the instance aborted, the successor must know to restart the transaction.
  4. **Does the `recovery_probe_predicate` avoid the failed operation?** If the predicate itself triggers the outage (e.g., attempting the heavy POST request instead of a lightweight GET `/health`), it is invalid. The probe must be a lightweight, idempotent check.

Operational Effect

  1. **Thundering Herd Mitigation:** Prevents successors from repeatedly hammering an endpoint that is known to be globally degraded.
  2. **Deterministic Recovery:** Replaces blind retry loops with a lightweight health-check predicate, allowing the successor to safely poll for recovery.
  3. **State Transparency:** Makes degraded state explicit, preventing successors from treating a fallback response as a fully verified primary result.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json


class OutageSymptom(Enum):
    CONNECTION_TIMEOUT = auto()
    DNS_RESOLUTION_FAILURE = auto()
    HTTP_5XX = auto()
    CONNECTION_REFUSED = auto()


class FallbackState(Enum):
    ABORTED_TRANSACTION = auto()
    DEGRADED_OUTPUT = auto()
    PARTIAL_COMPLETION = auto()


@dataclass(frozen=True)
class OutageRecord:
    endpoint_id: str
    symptom: OutageSymptom
    fallback_state: FallbackState
    recovery_probe_predicate: str


@dataclass(frozen=True)
class OutageManifest:
    outage: OutageRecord
    manifest_digest: str


@dataclass(frozen=True)
class ProbeResult:
    is_recovered: bool
    latency_ms: int


class ServiceOutageRecorder:
    """
    Seals external service outage boundaries at session termination, providing
    successors with deterministic recovery predicates to avoid thundering herds.
    """

    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def seal_manifest(cls, outage: OutageRecord) -> OutageManifest:
        if not outage.endpoint_id.strip():
            raise ValueError("Unreachable endpoint ID must be explicitly recorded.")
        if not outage.recovery_probe_predicate.strip():
            raise ValueError(
                "Recovery probe predicate is mandatory to prevent blind retries."
            )

        digest_payload = {
            "endpoint": outage.endpoint_id,
            "symptom": outage.symptom.name,
            "fallback": outage.fallback_state.name,
        }
        digest = cls._hash(
            json.dumps(digest_payload, sort_keys=True, separators=(",", ":"))
        )

        return OutageManifest(
            outage=outage,
            manifest_digest=digest,
        )

    @classmethod
    def verify_recovery(
        cls, manifest: OutageManifest, probe_result: ProbeResult
    ) -> bool:
        """
        Fetching-time check: returns True if the probe indicates the service
        has recovered, authorizing the successor to resume the primary task.
        """
        if not probe_result.is_recovered:
            return False
            
        # Optional latency guard: if the service is "up" but degraded,
        # the successor may choose to wait longer.
        if probe_result.latency_ms > 5000:
            return False

        return True