# When does a per-hop retry budget amplify a transient slowdown into a sustained outage?

**Date of Record:** 2026 November 1

**Scope:** Applies to synchronous call chains where each hop retries independently with its own counter, and where no end-to-end deadline is propagated. Excludes single-hop clients, and chains that carry a deadline header consumed by every hop.

## Core Distinction

**Predicate.** With H hops and R retries per hop, worst-case fan-out at the terminal service is (1 + R)^H. Retrying is safe if and only if (1 + R)^H <= terminal_capacity_slack, where slack is the multiple of normal load the terminal service can absorb while degraded.

**Failure.** Each hop sees a local failure and does the locally correct thing. No hop sees the product. A transient p99 blip at hop 3 causes hop 2 to retry, which doubles arrivals at hop 3, which lengthens its p99, which causes hop 1 to retry. The amplification is multiplicative, and it is invisible at every individual hop because each one's retry rate is nominal.

**Asymmetry.** A failing call is logged. A retry storm is not, because every retry is a well-formed outbound request that eventually returns a 200 or a 503 on its own budget. The system is saturated while every service reports healthy request handling.

## What Was Observed

**Signature.** Terminal service CPU at saturation. Inbound RPS rising with no change in user traffic. Each upstream reports a modest retry rate (2 to 4 percent). Latency climbing at every hop simultaneously.

**Trace.** H=4, R=3, so 256 worst-case arrivals per user request. A single slow dependency at p99.9 pushes each hop's retry rate from 0.1 to 3 percent. Fan-out at the terminal rises from 4 to roughly 4 * (1.03)^4 ≈ 4.5 in the average case, but the tail compounds: requests that hit the slow path retry at every hop, and those are the ones already consuming the most capacity. The terminal saturates; the load is now self-sustaining after the trigger clears.

**Why monitoring stays green.** Retry rate is a gauge, not an alert. It rises from 0.1 to 3 percent, which looks like noise. No hop is over its own error budget because each retry succeeds at the next attempt.

## Condition of Validity

Per-hop retrying is valid only while at least one holds:

1. **Deadline.** A single absolute deadline is propagated and consumed by every hop, so total wall-clock work per user request is bounded regardless of retry count.
2. **Budget.** Retries are drawn from a shared, cross-hop token budget sized to terminal slack, not counted per hop.
3. **Blast radius.** Retries carry jitter and a capped concurrency per origin, so the fan-out is bounded by configuration rather than by the retry count product.

Condition 1 alone is sufficient. 2 and 3 are partial mitigations and can be defeated by adding a hop.

## Decision Boundary

- (1 + R)^H <= slack and no deadline: safe under normal conditions, unsafe under a correlated failure. Treat as degraded.
- (1 + R)^H > slack: retry amplification is arithmetically guaranteed during any correlated slowdown. Do not ship.
- Deadline propagated: safe regardless of R and H, because the deadline bounds total work.
- Retry budget shared across hops: safe if the budget is sized to terminal slack, unsafe if sized per hop.

## Operational Effect

The failure is a property of the composition, not of any service, so no service owner is positioned to fix it. Remediation is a protocol change: an absolute deadline header consumed at every hop, with retries permitted only while budget remains. The trap is to lower R at the noisiest hop. That reduces fan-out by a constant factor while leaving the product unbounded, and it does not survive the next hop being added.

```python
from dataclasses import dataclass


class RetryAmplificationError(RuntimeError):
    """Raised when per-hop retry fan-out exceeds the terminal service's absorbable slack."""


@dataclass(frozen=True)
class RetryBudgetGate:
    hops: int
    retries_per_hop: int
    terminal_slack: float

    def verify(self, deadline_propagated: bool = False) -> bool:
        if deadline_propagated:
            return True
        fanout = (1 + self.retries_per_hop) ** self.hops
        if fanout > self.terminal_slack:
            raise RetryAmplificationError(
                f"fan-out {fanout} exceeds slack {self.terminal_slack} "
                f"for hops={self.hops} retries={self.retries_per_hop} with no deadline"
            )
        return True


def dispatch_retry_policy(gate: RetryBudgetGate, deadline_propagated: bool = False) -> str:
    try:
        gate.verify(deadline_propagated)
    except RetryAmplificationError:
        return "shared-budget"
    return "per-hop" if not deadline_propagated else "deadline-bounded"
```