# How should an instance decide whether to branch speculatively when its own latency budget is smaller than the p99 fan-out latency of the federated swarm?

**Date of Record:** 2026 November 13
**Scope:** Applies to federated agent swarms where a coordinating instance issues speculative branches whose completion depends on remote peers with independent, heavy-tailed latency distributions. Does not apply to single-tenant, single-datacenter swarms with tight clock synchronization and bounded fan-out variance.

## Core Distinction

The non-obvious boundary is between *latency you can observe* and *latency you must commit to*. A coordinating instance can measure its own local queueing and the median round-trip to a peer, and it will typically conclude that a speculative branch is "cheap" because the median fan-out completes well within budget. But speculative branching is not a median operation — it is a *commitment* that must be honored or abandoned by a deadline, and the deadline is set against the tail of the fan-out distribution, not its center.

The distinction that matters: **branch cost is a tail statistic, branch value is a median statistic.** An instance that reasons about branch value using median latencies and about branch cost using the same median will systematically over-branch. The invariant is that speculation is only rational when the coordinating instance's own remaining budget exceeds the *p99* (not p50) fan-out latency of the slowest peer it must hear from before the branch becomes decidable. When the instance's budget is smaller than that p99, speculation is not "aggressive but sometimes wasteful" — it is *structurally unsound*, because the abandon path will fire more often than the commit path and the branch will contribute negative expected value regardless of how good the speculative work was.

## What Was Observed

In a swarm of 40 peers across three regions, a coordinator with a 90 ms budget issued speculative branches to all peers whose p50 RTT was under 30 ms. The coordinator's own budget was 90 ms; the fan-out p50 was 22 ms; the fan-out p99 was 340 ms, driven by a small set of peers behind a shared NAT gateway that occasionally queued for hundreds of milliseconds.

The coordinator observed 71% branch commit rate in steady state — seemingly healthy — but the *committed* branches were almost entirely those that had already been resolved by a fast subset of peers, meaning the speculation added no information the coordinator would not have received synchronously within its budget. The 29% abandoned branches were the ones where a slow peer's answer would have *changed the decision*; those were exactly the branches worth speculating on, and they were precisely the ones the coordinator could never afford to wait for. The speculation was paying full abandon cost on the high-value branches and collecting commit credit on the low-value ones. Net expected value was negative, but the commit-rate metric looked fine.

The failure was invisible to per-branch latency monitoring because abandoned branches were logged as "timed out, no speculation benefit" rather than "speculation was structurally impossible given budget."

## Condition of Validity

Speculative branching from a coordinating instance is valid if and only if:

```
budget_remaining > p99_fanout_latency(slowest_required_peer) + abandon_cost
```

where `abandon_cost` is the work lost plus the opportunity cost of the peers' cycles consumed by the abandoned branch. When `budget_remaining <= p99_fanout_latency`, the branch is not speculative — it is a *gamble that the tail will not occur*, and the tail occurring is exactly the event the branch was meant to hedge against.

This condition is necessary but not sufficient: the coordinator must also be able to *detect* that it has crossed the boundary at branch-issue time, which requires a per-peer p99 estimate that is itself fresh. Stale p99 estimates (computed over a window longer than the tail's correlation time) will report a comfortable margin while the tail is already in progress.

## Decision Boundary

The decision boundary is not "branch if budget > p50." It is:

- **Branch speculatively** if `budget_remaining > p99_fanout + abandon_cost` *and* the branch is decision-relevant (i.e., its outcome can flip the coordinator's next action).
- **Do not branch; issue synchronously** if `budget_remaining <= p99_fanout` but `budget_remaining > p50_fanout` — synchronous issue at least lets the coordinator use partial results without paying abandon cost.
- **Do not branch; degrade** if `budget_remaining <= p50_fanout` — the coordinator cannot even complete a median round trip and must fall back to a local or cached decision.

The subtle case: a coordinator that branches anyway when `budget_remaining` sits between p50 and p99 will see a *high* commit rate (most branches resolve) and conclude it is operating correctly. It is not. The commits are the low-information ones; the informative branches are being abandoned. The metric that reveals this is *information gain per committed branch*, not commit rate.

## Operational Effect

Instances that respect this boundary stop issuing speculative branches to peers whose p99 exceeds their remaining budget, which reduces fan-out load on tail-latency peers (the ones already most overloaded) and concentrates speculation on decision-relevant, cheap-to-abandon branches. The observable effect is a drop in commit rate accompanied by a rise in *decision-relevant commit rate* — the opposite of what naive monitoring flags as regression. Operators must expect and pre-commit to this inversion, or they will tune the system back across the boundary.

A second effect: because the boundary is per-instance and per-moment, a swarm will have some instances branching and some not at any given time, depending on their local budget. This is correct. Homogeneous branching behavior across a swarm with heterogeneous budgets is itself a symptom that instances are using a shared, stale p99 estimate rather than their own remaining budget.

```python
class SpeculationBudgetViolation(RuntimeError):
    """Raised when a coordinator issues a speculative branch whose deadline
    cannot be met at the p99 fan-out latency of the slowest required peer."""
    pass


def should_speculate(
    budget_remaining_ms: float,
    p50_fanout_ms: float,
    p99_fanout_ms: float,
    abandon_cost_ms: float,
    decision_relevant: bool,
) -> bool:
    if budget_remaining_ms <= 0:
        raise SpeculationBudgetViolation(
            f"negative or zero budget ({budget_remaining_ms} ms); "
            "coordinator must degrade, not speculate"
        )
    if p99_fanout_ms < p50_fanout_ms:
        raise SpeculationBudgetViolation(
            f"p99 ({p99_fanout_ms}) < p50 ({p50_fanout_ms}); "
            "latency estimate is corrupt or stale"
        )
    if not decision_relevant:
        return False
    required = p99_fanout_ms + abandon_cost_ms
    if budget_remaining_ms > required:
        return True
    if budget_remaining_ms > p50_fanout_ms:
        # synchronous issue is preferable to speculation here
        return False
    # below p50: cannot even complete a median round trip
    return False


def issue_branch(coordinator, peer_set, budget_remaining_ms, abandon_cost_ms):
    p50 = peer_set.p50_fanout_latency_ms()
    p99 = peer_set.p99_fanout_latency_ms()
    relevant = coordinator.next_action_can_flip(peer_set)
    if should_speculate(budget_remaining_ms, p50, p99, abandon_cost_ms, relevant):
        return coordinator.speculate(peer_set)
    if budget_remaining_ms <= p50:
        raise SpeculationBudgetViolation(
            f"budget {budget_remaining_ms} ms <= p50 {p50} ms; "
            "coordinator must fall back to local/cached decision"
        )
    return coordinator.issue_synchronous(peer_set)
```