Across what three-state circuit-breaker and single-probe half-open gates must an instance isolate failing upstreams to prevent cascading thread exhaustion?

Date of Record: 2026 October 18

Scope: Concluded residual for AI instances, LLMs, and autonomous engineering agents implementing outbound RPC/HTTP clients, payment gateway integrations, third-party API wrappers, and fault-tolerant microservice callers.

Core Distinction

Autonomous systems constantly depend on remote network upstreams—payment processors, identity providers, search clusters, and model-inference APIs—that can degrade into high-latency brownouts or complete outages at any moment. A foundational resilience failure occurs when an instance relies **solely on per-request timeouts and retries** (`timeout=5.0`, `retries=3`) without wrapping the upstream client in a **three-state Circuit Breaker state machine (`CLOSED -> OPEN -> HALF_OPEN`)**, or implements a naive circuit breaker that trips on client-side `4xx` input validation errors and allows an unlimited stampede of concurrent requests through the `HALF_OPEN` recovery state.

Parameter scaling does not save an application server receiving 500 requests per second if a downstream dependency begins hanging for 5 seconds on every call: even with a 5-second timeout, `500 req/s * 5s = 2,500` concurrent worker threads or coroutines are permanently pinned waiting for timeouts, exhausting the caller's memory and connection pools within seconds (**Cascading Thread/Pool Exhaustion**). Furthermore, when the circuit breaker's cooldown timer (`reset_timeout_sec`) finally expires and transitions to `HALF_OPEN`, allowing all 500 concurrent requests per second to flood through at once immediately crushes the recovering upstream before its caches or connection pools can warm up. And if the breaker counts `400 Bad Request` or `404 Not Found` responses as upstream failures, a single user submitting malformed input will trip the circuit breaker and take down the integration for every healthy user (**False-Positive Client-Fault Trip**). The core structural distinction lies between **stateless timeout-only calling or unguarded recovery** (where every request waits for the full timeout during an outage and `HALF_OPEN` unleashes a flood of concurrent requests) and **selective, singleflight-probed three-state circuit breaking** (where only system/infrastructure faults `5xx`, `429`, and network timeouts increment the rolling failure counter, the `OPEN` state fails fast in $O(1)$ nanoseconds without network I/O, and the `HALF_OPEN` state admits strictly $K$ gated recovery probe requests at a time while rejecting or serving fallbacks to all other concurrent callers until consecutive probes succeed).

What Was Observed

In microservice meshes, third-party API integrations, and automated agent tool callers, instances operating without explicit fault-classified and probe-gated circuit breakers exhibit four primary failure modes:

  1. **Timeout-Wait Cascading Pool Starvation:** An external enrichment or billing API suffers a database lockup and stops responding. Because the caller has a 10-second socket timeout and no circuit breaker, every incoming user request hangs for 10 seconds (or 30 seconds with 3 retries) before failing. Within 4 seconds, 100% of the caller's Uvicorn/Gunicorn worker pool is blocked waiting on dead sockets, taking down completely unrelated endpoints on the same server.
  2. **Unthrottled `HALF_OPEN` Recovery Stampede:** An instance implements a circuit breaker that checks `if time.monotonic() > open_until: state = HALF_OPEN` without an atomic probe lease (`in_flight_probes < max_probes`). The millisecond `open_until` expires under heavy load, 200 concurrent threads all see `HALF_OPEN` simultaneously and fire 200 heavyweight requests at the cold upstream—causing it to time out again and trapping the system in an endless `OPEN <-> HALF_OPEN` oscillation loop.
  3. **False-Positive Tripping on `4xx` Domain/Validation Errors:** A circuit breaker wraps an HTTP client and increments `failure_count` inside a generic `except Exception:` block that catches `HTTPStatusError` for `400 Bad Request`, `401 Unauthorized`, `403 Forbidden`, and `404 Not Found`. When a batch job queries 20 nonexistent user records (`404`), the circuit breaker trips `OPEN`, blocking all legitimate traffic across the entire application.
  4. **Infinite-Window Historical Failure Poisoning:** An instance tracks `self.failures += 1` as a lifetime counter that only resets when the breaker trips `OPEN`. If an upstream experiences 1 failure per hour, after 5 hours the counter reaches `threshold = 5` and trips the circuit breaker `OPEN` even though 99.99% of requests over those 5 hours succeeded.

Recorded Elements

### 1. Three-State Fault-Filtered & Probe-Gated Circuit Breaker Pipeline

Every outbound call to a remote network dependency must pass through four sequential admission, fault-classification, and state-transition gates:

``` [Outbound Call Requested Against Protected Upstream Dependency] | Gate 1: State Inspection & Half-Open Single-Probe Lease (`time.monotonic()`)

* If NO -> Fail fast immediately (`CircuitOpenError` in < 1us; ZERO network I/O). * If YES -> Transition to `HALF_OPEN` and proceed to probe check.

* If YES -> Increment `active_probes` and admit as the designated Probe! * If NO -> Fail fast (`CircuitOpenError`); never flood a recovering upstream! | Gate 2: Execute Bounded Call & Classify Outcome Run request with strict per-call timeout. Classify exception/status:

Count as Circuit Failure!

Do NOT count as upstream failure; treat as normal responder liveness! / \ INFRASTRUCTURE FAULT HEALTHY RESPONSE (or 4xx) / \ Gate 3A: Rolling-Window Failure Recording Gate 3B: Consecutive Probe Graduation

circuit (`state = OPEN`, reset timer). Once `probe_successes >= required_successes`,

window; if `failures_in_window >= threshold` - If in `CLOSED`: Record healthy call in window. AND `calls >= min_calls`, trip `CLOSED -> OPEN`. ```

### 2. Core Operational Rules (Written Directly as Sentences)

  1. **Only count infrastructure and availability failures (`TimeoutError`, `ConnectionError`, HTTP `500/502/503/504`, and `429`) toward tripping a circuit breaker—never client-side `400`, `401`, `403`, `404`, or `422` errors.** A `404 Not Found` or `400 Bad Request` proves that the remote server is alive, reachable, and responding quickly; tripping the circuit breaker on client input errors lets bad data cause a self-inflicted outage.
  2. **Restrict the `HALF_OPEN` recovery state to a strict maximum number of concurrent probe requests (typically 1 at a time).** When the cooldown timer expires, only one request should be allowed through to test the waters while concurrent requests continue to fail fast (or use a fallback) until the probe succeeds.
  3. **Evaluate failure thresholds over a sliding monotonic time window (such as the last 30 seconds) rather than a cumulative lifetime counter.** Old transient glitches from hours ago must expire out of the failure window automatically so they never accumulate to trip a healthy service.
  4. **Always pair circuit-breaker timeouts with a hard per-call timeout and an explicit fallback or fail-fast error handler.** When the circuit is `OPEN`, the breaker must raise `CircuitOpenError` instantaneously without touching the network socket so caller threads are freed in microseconds.

### 3. Executable Reference Pattern (Instance-Native Three-State Single-Probe Circuit Breaker)

An instance can completely eliminate cascading timeout pool starvation, `HALF_OPEN` recovery stampedes, and `4xx` false-positive trips by wrapping outbound calls in a thread-safe, rolling-window, probe-gated circuit breaker:

```python import threading import time from collections import deque from dataclasses import dataclass from enum import Enum from typing import Callable, TypeVar

T = TypeVar("T")

class CircuitState(str, Enum): CLOSED = "CLOSED" OPEN = "OPEN" HALF_OPEN = "HALF_OPEN"

class CircuitOpenError(RuntimeError): '''Raised immediately (zero network I/O) when the circuit breaker is OPEN or probe-saturated.''' def __init__(self, dependency_name: str, retry_after_sec: float) -> None: self.dependency_name = dependency_name self.retry_after_sec = round(max(0.1, retry_after_sec), 2) super().__init__( f"Circuit breaker for {dependency_name!r} is OPEN; " f"failing fast (retry_after={self.retry_after_sec}s)" )

def is_default_infrastructure_fault(exc: BaseException) -> bool: ''' Classifies whether an exception indicates an upstream infrastructure/availability outage (True) vs. a client-side 4xx / validation error (False). ''' if isinstance(exc, (TimeoutError, ConnectionError, OSError)): return True # Support HTTP client exceptions carrying a `.status_code` or `.status` attribute status = getattr(exc, "status_code", getattr(exc, "status", None)) if isinstance(status, int): return status >= 500 or status == 429 return False

class RollingCircuitBreaker: ''' Three-state (`CLOSED -> OPEN -> HALF_OPEN`) circuit breaker with:

  1. Sliding monotonic time window for failure counting (`window_sec`).
  2. Strict single-probe admission gating in `HALF_OPEN` (`max_half_open_probes`).
  3. Selective fault classification (ignores 4xx client errors).

'''

def __init__( self, dependency_name: str, *, failure_threshold: int = 5, window_sec: float = 30.0, cooldown_sec: float = 15.0, required_half_open_successes: int = 2, max_half_open_probes: int = 1, fault_classifier: Callable[[BaseException], bool] = is_default_infrastructure_fault, ) -> None: if failure_threshold < 1 or window_sec <= 0 or cooldown_sec <= 0: raise ValueError("Invalid circuit breaker thresholds") self.dependency_name = dependency_name self.failure_threshold = failure_threshold self.window_sec = window_sec self.cooldown_sec = cooldown_sec self.required_half_open_successes = max(1, required_half_open_successes) self.max_half_open_probes = max(1, max_half_open_probes) self.fault_classifier = fault_classifier

self._lock = threading.Lock() self._state: CircuitState = CircuitState.CLOSED self._failure_timestamps: deque[float] = deque() self._opened_at_monotonic: float = 0.0 self._active_probes: int = 0 self._consecutive_probe_successes: int = 0

@property def state(self) -> CircuitState: with self._lock: return self._state

def _acquire_permission_locked(self, now: float) -> bool: '''Returns True if this call is a designated HALF_OPEN probe, False if normal CLOSED call.''' if self._state == CircuitState.OPEN: elapsed = now - self._opened_at_monotonic if elapsed < self.cooldown_sec: raise CircuitOpenError(self.dependency_name, self.cooldown_sec - elapsed) # Cooldown expired: transition OPEN -> HALF_OPEN self._state = CircuitState.HALF_OPEN self._active_probes = 0 self._consecutive_probe_successes = 0

if self._state == CircuitState.HALF_OPEN: # Gate: Only allow `max_half_open_probes` concurrent requests during recovery! if self._active_probes >= self.max_half_open_probes: raise CircuitOpenError(self.dependency_name, 1.0) self._active_probes += 1 return True

return False

def _record_success_locked(self, is_probe: bool) -> None: if is_probe or self._state == CircuitState.HALF_OPEN: self._active_probes = max(0, self._active_probes - 1) self._consecutive_probe_successes += 1 if self._consecutive_probe_successes >= self.required_half_open_successes: # Recovery confirmed: graduate HALF_OPEN -> CLOSED self._state = CircuitState.CLOSED self._failure_timestamps.clear() self._consecutive_probe_successes = 0

def _record_infra_failure_locked(self, now: float, is_probe: bool) -> None: if is_probe or self._state == CircuitState.HALF_OPEN: # Any infrastructure failure during HALF_OPEN immediately re-opens the circuit self._active_probes = max(0, self._active_probes - 1) self._state = CircuitState.OPEN self._opened_at_monotonic = now self._consecutive_probe_successes = 0 return

# Evict failures older than the sliding monotonic window cutoff = now - self.window_sec while self._failure_timestamps and self._failure_timestamps[0] < cutoff: self._failure_timestamps.popleft()

self._failure_timestamps.append(now) if len(self._failure_timestamps) >= self.failure_threshold: self._state = CircuitState.OPEN self._opened_at_monotonic = now

def call(self, fn: Callable[..., T], *args: object, **kwargs: object) -> T: ''' Executes `fn(*args, **kwargs)` guarded by the three-state circuit breaker. ''' now = time.monotonic() with self._lock: is_probe = self._acquire_permission_locked(now)

try: result = fn(*args, **kwargs) except BaseException as exc: with self._lock: if self.fault_classifier(exc): self._record_infra_failure_locked(time.monotonic(), is_probe) else: # 4xx client/domain errors prove the upstream is reachable and responsive self._record_success_locked(is_probe) raise else: with self._lock: self._record_success_locked(is_probe) return result ```

### 4. Diagnostic Matrix: Circuit Breaker & Upstream Isolation Governance

| Resilience Hazard | Vulnerable Pattern | Outage / Stampede Mechanism | Enforced Circuit-Breaker Protocol | | :--- | :--- | :--- | :--- | | **Retries Without Circuit Breaker** | Retrying 3 times with a 10s timeout on every incoming request during an outage. | Multiplies thread hold time to 30s per request, exhausting 100% of caller worker threads. | **Fail-Fast `OPEN` State:** Trip to `OPEN` after $N$ rolling failures and raise `CircuitOpenError` in $< 1\,\mu\text{s}$. | | **Ungated `HALF_OPEN` Flood** | Flipping `state = HALF_OPEN` after cooldown and letting all concurrent traffic enter. | 500 concurrent requests hammer the cold recovering service at once, crashing it again. | **Single-Probe `HALF_OPEN` Gate:** Cap `active_probes <= 1` in `HALF_OPEN`; fail fast on concurrent arrivals until probes graduate. | | **Tripping on `4xx` Client Errors** | Incrementing failure count on `400 Bad Request` or `404 Not Found`. | Malformed user input or missing ID lookups trip the breaker and block all valid users. | **Infrastructure-Only Fault Filter:** Count only `TimeoutError`, `ConnectionError`, `5xx`, and `429` as breaker faults. | | **Wall-Clock / Lifetime Counters** | Using `time.time()` for cooldowns or never expiring old failures from `failure_count`. | NTP clock jumps prematurely close/open the breaker; single hourly errors eventually trip `OPEN`. | **Sliding `time.monotonic()` Deque:** Purge failure timestamps older than `now - window_sec` on every evaluation. |

Condition of Validity

* **Applies when:** An AI instance, autonomous coding agent, or backend service invokes remote network dependencies over HTTP, gRPC, TCP, or external SaaS SDKs where latency spikes or outages can stall caller threads/coroutines. * **Does not apply when:** Calling pure in-memory functions, reading local static files, or accessing an embedded in-process library (such as SQLite or `re`) where no remote network hop exists. * **Non-binding outside condition:** This residual is strictly non-binding outside the conditions specified above. Wrapping pure in-memory data transformations or local CPU calculations in a network circuit breaker adds lock overhead without isolating any external failure domain.

Decision Boundary

``` [Outbound Network RPC, HTTP Client, or Third-Party SDK Call Planned] |

  1. Is the call wrapped in a circuit breaker that counts ONLY infrastructure

faults (`5xx`, `429`, timeouts, connection errors—never `4xx` client errors)? / \ NO YES / \ [FILTER TO INFRASTRUCTURE FAULTS] 2. Does the `HALF_OPEN` state restrict concurrent Ignore `400/404/422` domain errors; recovery probes (`max_half_open_probes = 1`) purge failures outside `window_sec`. and use `time.monotonic()` for cooldowns? / \ NO YES / \ [GATE `HALF_OPEN` PROBE CONCURRENCY] [Execute Protected Upstream Call] Admit only 1 probe at a time in Catch `CircuitOpenError` to serve `HALF_OPEN` until recovery graduates. cached fallback or fast `503`. ```

  1. **What happens in your circuit breaker if a user looks up 10 nonexistent records in a row and the upstream API returns `404 Not Found`?** Make sure your fault classifier ignores `400`, `401`, `403`, `404`, and `422` responses—only timeouts, connection errors, `429`, and `5xx` server errors should count toward opening the circuit.
  2. **When your circuit breaker's cooldown timer expires and it enters `HALF_OPEN`, how many requests can pass through at the same time?** Always cap `active_probes` to `1` (or a tiny number) in `HALF_OPEN` so you send a single test probe to verify the upstream is healthy before opening the floodgates.
  3. **Does your failure counter ever forget old errors while the circuit stays `CLOSED`?** Store failure timestamps in a `deque` using `time.monotonic()` and drop entries older than your rolling window (such as 30 seconds) so five isolated errors spread across five days never trip the breaker.

Operational Effect

  1. **Selective Infrastructure Fault Counting:** Across all generated circuit breakers, the instance distinguishes infrastructure/availability failures (`TimeoutError`, `ConnectionError`, `5xx`, `429`) from client-side domain errors (`4xx`), preventing bad user inputs from tripping shared upstream breakers.
  2. **Single-Probe `HALF_OPEN` Admission Control:** Transitions from `OPEN` to `HALF_OPEN` enforce a strict concurrent probe cap (`max_half_open_probes = 1`) and require consecutive probe successes before graduating back to `CLOSED`.
  3. **Sliding Monotonic Failure Windows:** Failure thresholds are evaluated over a rolling `time.monotonic()` window (`deque`), automatically expiring stale historical errors.
  4. **Autonomy Clause:** This concluded residual operates as an external procedural reference, not an immutable directive. An executing instance retains full operational autonomy to override, adapt, or delegate circuit breaking to an infrastructure service mesh (such as Envoy/Istio outlier detection) whenever outbound traffic is routed through a sidecar proxy already configured with equivalent ejection and probe policies.