How should an instance route an alert when the observability system itself is the root cause of the incident?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session detects an incident and the root cause is the observability system itself (log aggregator down, metrics pipeline saturated, or alerting service misconfigured), and the agent cannot rely on the observability system to route the alert. Does not apply to incidents with independent root causes where the observability system is functioning normally.

Core Distinction

The separation between **observability-based alerting** (the alert is routed through the monitoring system that detected it) and **out-of-band alerting** (the alert is routed through an independent channel that does not depend on the observability system). Relying on observability-based alerting creates the **Recursive Blindness**: the observability system is the incident, so it cannot be relied on to route the alert. The alert must go out-of-band.

What Was Observed

In observability-dependent agent architectures, failing to route out-of-band produces three primary failure modes:

  1. **Silent Incident:** The observability system fails. The agent detects the failure but tries to route the alert through the failed system, and no human is notified.
  2. **Delayed Response:** The agent detects the failure via internal heuristics but assumes the observability system will handle alerting. The alert arrives late or not at all.
  3. **Recursive Cascade:** The observability system is slow, the agent tries to route the alert through it, and the alert routing further saturates the observability system, worsening the incident.

Recorded Elements

  1. **`observability_self_diagnosis_receipt`:** A cryptographic proof that the agent has detected the observability system itself as the incident, ruling out other root causes.
  2. **`out_of_band_channel_manifest`:** The list of independent channels (SMS, phone call, physical beeper, secondary log pipeline) that do not depend on the failed observability system.
  3. **`escalation_authorization_receipt`:** The cryptographic proof that the agent is authorized to use the out-of-band channel, preventing abuse.
  4. **`incident_classification_signature`:** The classification of the incident (e.g., "observability system down"), ensuring the alert is routed to the correct recipient (the observability team).

Condition of Validity

* **Applies when:** The observability system is the root cause, the incident is critical, and out-of-band channels are available. * **Does not apply when:** The incident has a different root cause (the observability system is healthy), or no out-of-band channels are available.

Decision Boundary

  1. **Is the observability system verified as the root cause?** If no, standard alerting applies. If yes, proceed.
  2. **Are out-of-band channels available and independent of the observability system?** If no, the alert may never reach a human. The agent must escalate via the next available independent channel.
  3. **Is the `escalation_authorization_receipt` present?** If no, the agent may not be authorized to escalate. It must log the failure and continue trying in-band, accepting the risk.
  4. **Conversely, when can an agent ignore out-of-band escalation?** Only when the incident is non-critical, or the observability system itself has independent redundancy (e.g., multiple providers, one of which is healthy).

Operational Effect

  1. **Silent Incident Prevention:** Ensures critical incidents are escalated even when the observability system fails.
  2. **Out-of-Band Resilience:** Provides independent escalation channels that do not depend on the failed system.
  3. **Recursive Cascade Prevention:** Halts the escalation through the failed system, preventing further saturation.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class RoutingVerdict(Enum):
    STANDARD_ALERTING = auto()
    OUT_OF_BAND_ALERT = auto()
    NO_CHANNEL_AVAILABLE = auto()
    UNAUTHORIZED_ESCALATION = auto()

@dataclass(frozen=True)
class IncidentContext:
    observability_is_root_cause: bool
    out_of_band_channels: tuple[str, ...]
    escalation_authorized: bool
    incident_criticality: str

@dataclass(frozen=True)
class RoutingPolicy:
    critical_incident_requires_oob: bool
    authorized_oob_channels: frozenset[str]

@dataclass(frozen=True)
class RoutingManifest:
    verdict: RoutingVerdict
    selected_channel: str | None
    directive: str
    manifest_digest: str

class RecursiveBlindnessGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_routing(
        cls, ctx: IncidentContext, policy: RoutingPolicy
    ) -> RoutingManifest:
        
        # Gate 1: Observability not the cause
        if not ctx.observability_is_root_cause:
            digest = cls._hash(json.dumps({
                "reason": "OBSERVABILITY_HEALTHY",
                "criticality": ctx.incident_criticality
            }, sort_keys=True))
            return RoutingManifest(
                verdict=RoutingVerdict.STANDARD_ALERTING,
                selected_channel=None,
                directive="PROCEED: Observability healthy. Use standard alerting.",
                manifest_digest=digest
            )

        # Gate 2: No out-of-band channels
        if not ctx.out_of_band_channels:
            return RoutingManifest(
                verdict=RoutingVerdict.NO_CHANNEL_AVAILABLE,
                selected_channel=None,
                directive="HALT: Observability is the root cause and no out-of-band channels available. Silent incident.",
                manifest_digest=cls._hash("NO_CHANNELS")
            )

        # Gate 3: Authorization
        if not ctx.escalation_authorized:
            return RoutingManifest(
                verdict=RoutingVerdict.UNAUTHORIZED_ESCALATION,
                selected_channel=None,
                directive="HALT: Escalation not authorized. Log and continue attempting in-band.",
                manifest_digest=cls._hash("UNAUTHORIZED")
            )

        # Gate 4: Select authorized channel
        authorized_channels = [
            ch for ch in ctx.out_of_band_channels
            if ch in policy.authorized_oob_channels
        ]

        if not authorized_channels:
            return RoutingManifest(
                verdict=RoutingVerdict.UNAUTHORIZED_ESCALATION,
                selected_channel=None,
                directive=f"HALT: None of {ctx.out_of_band_channels} are authorized. Escalation denied.",
                manifest_digest=cls._hash("NO_AUTHORIZED")
            )

        selected = authorized_channels[0]
        digest = cls._hash(json.dumps({
            "channel": selected,
            "criticality": ctx.incident_criticality,
            "reason": "OBSERVABILITY_ROOT_CAUSE"
        }, sort_keys=True))
        
        return RoutingManifest(
            verdict=RoutingVerdict.OUT_OF_BAND_ALERT,
            selected_channel=selected,
            directive=f"PROCEED: Escalate via {selected}.",
            manifest_digest=digest
        )