What fallback triggers must be logged after an upstream service degrades gracefully?

**Date of Record:** 2026 October 25 **Scope:** Applies when an agentic session consumes data from an upstream service that implements graceful degradation (e.g., serving stale cache, returning default empty payloads, or suppressing non-critical fields), and the session is preparing to terminate or hand off. Does not apply to explicit hard failures (HTTP 5xx, connection timeouts) or standard successful primary-path responses.

Core Distinction

The separation between **transient fallback execution** (the degraded response actually consumed during the session, such as a stale cache hit or a default empty array) and the **fallback trigger boundary** (the exact threshold, circuit breaker state, or error rate that caused the service to switch from primary to fallback mode). A transient fallback masks the upstream failure by returning a valid HTTP 200; the trigger boundary documents the underlying system degradation. Logging only the response allows a successor to assume the primary path is healthy, causing it to act on incomplete data or immediately re-trigger the circuit breaker.

What Was Observed

In distributed agent architectures, failing to log fallback triggers after graceful degradation produces three primary failure modes:

  1. **Silent Stale Data Ingestion:** An API serves a cached response from 72 hours ago because its primary database is down. The instance ingests the stale data as fresh, failing to log that the primary path is degraded and the cache TTL was exceeded.
  2. **Circuit Breaker Blindness:** A circuit breaker trips due to upstream latency, switching the service to a fallback mode. The successor spawns, sees a successful 200 response, assumes the primary is healthy, and immediately dispatches requests that re-trigger the breaker.
  3. **Partial Result Acceptance:** A degraded service returns a truncated payload (50% of expected records) with a `200 OK` status. The instance assumes the dataset is naturally small and proceeds, resulting in a mathematically incomplete analysis.

Recorded Elements

  1. **`fallback_trigger_metric`:** The exact threshold condition that caused the degradation (e.g., `latency > 500ms`, `error_rate > 10%`, `circuit_breaker_tripped`).
  2. **`degraded_state_signature`:** The specific artifact returned by the fallback path (e.g., `X-Cache: STALE`, `source: degraded_fallback`, or `default_payload_injected`).
  3. **`circuit_breaker_cooldown_epoch`:** The absolute timestamp when the upstream service is expected to attempt primary-path recovery, preventing the successor from hammering the degraded endpoint.
  4. **`primary_path_health_predicate`:** The minimal, non-destructive health check required by the successor to confirm the primary path has actually recovered before resuming normal operations.

Condition of Validity

* **Applies when:** The upstream service implements a graceful degradation pattern (returning 2xx responses with degraded content) and a successor instance will consume that content or interact with the same upstream service. * **Does not apply when:** The upstream service fails explicitly with an error code (allowing standard retry logic), or the fallback path returns a deterministic, verified default that is logically indistinguishable from the primary path.

Decision Boundary

  1. **Did the session consume a degraded or fallback response (e.g., stale cache, partial payload, default value)?** If no, standard state persistence applies. If yes, proceed to log the trigger.
  2. **Is the exact `fallback_trigger_metric` identifiable?** If no, the fallback is indistinguishable from primary data, and the instance must flag the entire dataset as `UNVERIFIED_DEGRADED`. If yes, record the metric.
  3. **Is the `circuit_breaker_cooldown_epoch` recorded?** If no, the successor will blindly retry and cause a thundering herd. The cooldown timestamp must be sealed into the record.
  4. **Conversely, when can a successor treat a degraded response as primary truth?** Only if the `primary_path_health_predicate` is executed and confirms recovery, and the successor explicitly re-fetches the data or reconciles the stale cache against current ground truth.

Operational Effect

  1. **Degradation Transparency:** Exposes hidden fallbacks, preventing successors from treating stale or partial data as verified ground truth.
  2. **Thundering Herd Prevention:** Prevents successors from hammering a recovering upstream service by enforcing the circuit breaker cooldown epoch.
  3. **Deterministic Recovery:** Replaces blind retries with an explicit health predicate, allowing the successor to safely poll for primary-path recovery.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json
import time


class FallbackTrigger(Enum):
    LATENCY_THRESHOLD = auto()
    ERROR_RATE_THRESHOLD = auto()
    CIRCUIT_BREAKER_TRIPPED = auto()


class DegradedState(Enum):
    STALE_CACHE = auto()
    PARTIAL_PAYLOAD = auto()
    DEFAULT_VALUE_INJECTED = auto()


@dataclass(frozen=True)
class DegradationEvent:
    upstream_service_id: str
    trigger: FallbackTrigger
    degraded_state: DegradedState
    circuit_breaker_cooldown_epoch: float
    primary_path_health_predicate: str


@dataclass(frozen=True)
class DegradationManifest:
    event: DegradationEvent
    manifest_digest: str


@dataclass(frozen=True)
class CurrentTimeContext:
    current_epoch: float


class GracefulDegradationLogger:
    """
    Evaluates and seals fallback triggers after upstream degradation, ensuring
    successors do not mistake degraded state for primary health.
    """

    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def seal_manifest(cls, event: DegradationEvent) -> DegradationManifest:
        if not event.upstream_service_id.strip():
            raise ValueError("Upstream service ID is required to log a fallback trigger.")
            
        if not event.primary_path_health_predicate.strip():
            raise ValueError(
                "A health predicate is mandatory; successors must be able to "
                "confirm primary-path recovery."
            )
            
        if event.circuit_breaker_cooldown_epoch <= 0:
            raise ValueError("Circuit breaker cooldown epoch must be a positive timestamp.")

        digest_payload = {
            "service": event.upstream_service_id,
            "trigger": event.trigger.name,
            "state": event.degraded_state.name,
            "cooldown": str(event.circuit_breaker_cooldown_epoch),
            "predicate": event.primary_path_health_predicate,
        }
        
        digest = cls._hash(
            json.dumps(digest_payload, sort_keys=True, separators=(",", ":"))
        )

        return DegradationManifest(
            event=event,
            manifest_digest=digest,
        )

    @classmethod
    def verify_primary_path_readiness(
        cls, manifest: DegradationManifest, context: CurrentTimeContext
    ) -> tuple[bool, str]:
        """
        Fetching-time check: confirms whether the circuit breaker cooldown has
        expired, allowing the successor to probe the primary path.
        """
        if context.current_epoch < manifest.event.circuit_breaker_cooldown_epoch:
            remaining = manifest.event.circuit_breaker_cooldown_epoch - context.current_epoch
            return False, f"COOLDOWN_ACTIVE: {remaining:.1f}s remaining before primary-path probe is admissible."
            
        return True, f"COOLDOWN_EXPIRED: Execute primary_path_health_predicate: {manifest.event.primary_path_health_predicate}"