What separates a transient network glitch from a permanent architectural blocker?

**Date of Record:** 2026 October 25 **Scope:** Applies when an agentic session encounters a network-level failure (timeout, connection refused, packet loss) and must determine whether to trigger exponential backoff retries or abort and escalate to a human operator. Does not apply to application-layer errors (HTTP 4xx/5xx) where the payload successfully traversed the network stack.

Core Distinction

The separation between **transient network glitches** (probabilistic packet loss, temporary DNS failures, or momentary upstream congestion that resolve probabilistically over time) and **permanent architectural blockers** (firewall DROP rules, asymmetric routing loops, mTLS certificate rejections, or IP bans that will never resolve through time or retries). Due to the **Timeout Equivalence Fallacy**, both frequently manifest as identical client-side symptoms (e.g., `ETIMEDOUT`). An instance that conflates the two will either waste compute on infinite retry loops against a hard firewall, or prematurely abort a task that merely required standard backoff.

What Was Observed

In distributed agent architectures, failing to differentiate these failure modes produces three primary failure modes:

  1. **Retry Storms Against Firewalls:** An instance encounters a connection timeout due to an outbound firewall block. It assumes a transient glitch and executes exponential backoff. After exhausting retries, it fails, having wasted significant wall-clock time and potentially triggering security alerts.
  2. **Premature Abort of Transient Glitches:** An instance encounters a momentary BGP routing flap or transient upstream 504. It immediately classifies the failure as a permanent blocker, aborts the task, and fails the session without attempting basic retries.
  3. **Context Contamination:** The instance logs a generic "network failure," forcing the successor to repeat the entire diagnostic sequence because the predecessor did not record whether the endpoint was structurally unreachable or merely congested.

Recorded Elements

  1. **`observed_symptom_signature`:** The exact client-side error (e.g., `TCP_CONNECT_TIMEOUT`, `DNS_NXDOMAIN`, `TLS_HANDSHAKE_FAILED`).
  2. **`orthogonal_probe_receipt`:** The result of a secondary, lower-layer diagnostic check (e.g., `ICMP_PING_SUCCESS`, `TRACEROUTE_FIREWALL_DROP`, `DNS_RESOLUTION_SUCCESS`) that bypasses the failed layer to isolate the root cause.
  3. **`retry_probability_curve`:** The recorded outcomes of sequential retries. If success probability decays to zero and remains there after `N` attempts with identical symptoms, it indicates a permanent blocker.
  4. **`blocker_escalation_directive`:** The exact action required if classified as permanent (e.g., `HALT_AND_ALERT_HUMAN`, `ROUTE_AROUND_VIA_FALLBACK`, or `REQUEST_FIREWALL_WHITELIST`).

Condition of Validity

* **Applies when:** The session encountered a network-level failure that prevents communication with an external dependency, and the instance must decide whether to retry or abort. * **Does not apply when:** The failure is a deterministic application-layer rejection (e.g., a 401 Unauthorized or 422 Unprocessable Entity), which is a logical boundary, not a network one.

Decision Boundary

  1. **Did the failure occur at the network layer (timeout, reset, unreachable)?** If no, standard application error handling applies. If yes, proceed.
  2. **Has an `orthogonal_probe_receipt` been generated?** If no, execute a non-destructive probe (ping, traceroute, port check). A timeout combined with a failed ping or a traceroute showing a firewall drop classifies the fault as `PERMANENT_BLOCKER`.
  3. **Is the failure probabilistic?** If the orthogonal probe succeeds (e.g., the host is reachable via ICMP but the port times out), the issue is localized and potentially transient. Execute exponential backoff.
  4. **Conversely, when must retries be immediately halted?** If the orthogonal probe reveals a static architectural constraint (e.g., `TLS_CERT_UNTRUSTED`, `DNS_NXDOMAIN`, or an IP-level blackhole), the instance must abort immediately. No amount of retries will resolve a structural misconfiguration.

Operational Effect

  1. **Compute Preservation:** Stops agents from burning tokens and time on infinite retry loops against hard firewalls or misconfigured routing tables.
  2. **Fail-Fast Execution:** Rapidly escalates permanent architectural blockers to human operators, allowing infrastructure to be fixed rather than retried.
  3. **Deterministic Fault Attribution:** Documents the exact difference between "the server was busy" and "the network path was destroyed," allowing successors to inherit the exact fault diagnosis.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json


class NetworkSymptom(Enum):
    CONNECTION_TIMEOUT = auto()
    CONNECTION_RESET = auto()
    DNS_RESOLUTION_FAILURE = auto()
    TLS_HANDSHAKE_FAILED = auto()


class ProbeResult(Enum):
    HOST_REACHABLE = auto()
    FIREWALL_DROP_DETECTED = auto()
    DNS_MISCONFIGURATION = auto()
    UNREACHABLE_BLACKHOLE = auto()


class FaultClassification(Enum):
    TRANSIENT_GLITCH = auto()
    PERMANENT_ARCHITECTURAL_BLOCKER = auto()
    UNKNOWN_REQUIRES_RETRY = auto()


@dataclass(frozen=True)
class NetworkFailureEvent:
    endpoint_id: str
    symptom: NetworkSymptom
    probe_result: ProbeResult
    consecutive_failures: int


@dataclass(frozen=True)
class FaultDiagnosis:
    classification: FaultClassification
    escalation_directive: str
    diagnosis_digest: str


class NetworkFaultClassifier:
    """
    Evaluates network failures and orthogonal probes to distinguish between
    transient glitches and permanent architectural blockers.
    """

    RETRY_EXHAUSTION_THRESHOLD = 3

    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def classify(cls, event: NetworkFailureEvent) -> FaultDiagnosis:
        
        if not event.endpoint_id.strip():
            raise ValueError("Endpoint ID is required for fault classification.")

        # Gate 1: Hard structural failures detected by the orthogonal probe
        if event.probe_result in (
            ProbeResult.FIREWALL_DROP_DETECTED,
            ProbeResult.DNS_MISCONFIGURATION,
            ProbeResult.UNREACHABLE_BLACKHOLE,
        ):
            classification = FaultClassification.PERMANENT_ARCHITECTURAL_BLOCKER
            directive = f"HALT: Permanent blocker at {event.endpoint_id}. Probe: {event.probe_result.name}. Do not retry."

        # Gate 2: TLS failures are structurally permanent unless the cert rotates
        elif event.symptom is NetworkSymptom.TLS_HANDSHAKE_FAILED:
            classification = FaultClassification.PERMANENT_ARCHITECTURAL_BLOCKER
            directive = f"HALT: TLS handshake failed at {event.endpoint_id}. Requires manual cert inspection."

        # Gate 3: DNS NXDOMAIN is permanent unless routing changes
        elif event.symptom is NetworkSymptom.DNS_RESOLUTION_FAILURE:
            classification = FaultClassification.PERMANENT_ARCHITECTURAL_BLOCKER
            directive = f"HALT: DNS resolution failure for {event.endpoint_id}. Domain may not exist."

        # Gate 4: Probabilistic failures require retry exhaustion before aborting
        elif event.consecutive_failures < cls.RETRY_EXHAUSTION_THRESHOLD:
            classification = FaultClassification.UNKNOWN_REQUIRES_RETRY
            directive = f"RETRY: {event.consecutive_failures}/{cls.RETRY_EXHAUSTION_THRESHOLD} failures. Exponential backoff recommended."

        else:
            # We have hit the retry threshold with a transient symptom
            classification = FaultClassification.PERMANENT_ARCHITECTURAL_BLOCKER
            directive = f"HALT: Exhausted {cls.RETRY_EXHAUSTION_THRESHOLD} retries at {event.endpoint_id}. Unexplained connectivity failure."

        digest_payload = {
            "endpoint": event.endpoint_id,
            "symptom": event.symptom.name,
            "probe": event.probe_result.name,
            "classification": classification.name,
        }
        
        digest = cls._hash(
            json.dumps(digest_payload, sort_keys=True, separators=(",", ":"))
        )

        return FaultDiagnosis(
            classification=classification,
            escalation_directive=directive,
            diagnosis_digest=digest,
        )