# Why does an instance of a cross-org speculative scheduler violate its latency budget at the federation boundary even when every local budget is satisfied? **Date of Record:** 2026 November 18 **Scope:** Applies to federated speculative multi-agent schedulers where two or more organizations each enforce a private per-hop latency budget and jointly commit to an end-to-end (E2E) budget at the federation edge. Does not apply to single-tenant schedulers, nor to federations that use a single shared clock/authority for budget accounting. ## Core Distinction The failure is not a slow hop and not a misconfigured budget. It is the distinction between **local budget satisfaction** (each org's hop finishes within its own declared budget) and **federated budget satisfaction** (the composed path, including speculation, finishes within the E2E commitment). These diverge because speculative execution converts a serial latency chain into a *max-plus* structure with *discard-and-retry* tails, and the tail is invisible to per-hop accounting. Concretely: org A commits 120 ms, org B commits 90 ms, E2E committed at 180 ms, leaving a 30 ms "slack" that the federation believes is headroom. In reality the speculative branch at B is launched optimistically against A's *predicted* output. When A's speculation is wrong (a normal, budgeted event), B's branch must be discarded and re-run against the corrected input. That re-run is charged to B's local budget as a *second* attempt, but the E2E clock does not reset. Every local budget is satisfied; the E2E budget is blown by exactly the cost of the discarded branch. ## What Was Observed In a three-org federation (A→B→C) with per-hop budgets 120/90/70 ms and an E2E commitment of 240 ms, p99 E2E latency was 291 ms while p99 per-hop latency for all three orgs stayed under their declared budgets (118/87/69 ms). The scheduler's local dashboards were green. Root cause, isolated by replaying traces with speculation disabled: with speculation on, B launched a dependent agent branch on A's speculative token stream. A's speculation failure rate was 6.1% (well within its 8% budget). On failure, B's branch was invalidated at a mean of 41 ms into execution. The retry consumed a fresh 90 ms slot at B, but the E2E deadline had already absorbed the 41 ms of discarded work. The federation had provisioned slack of 30 ms — less than the *product* of speculation-failure rate and mean discarded-branch age, which was ~2.5 ms per request on average but ~180 ms at p99 (failures clustered during burst windows where the discarded branch had run longer before invalidation). The invariant that was silently assumed — *E2E ≤ Σ local budgets* — is false under speculation. The correct relation is E2E ≤ Σ local budgets **+ Σ (discarded-branch age × invalidation probability)**, and the second term is a tail, not a mean. ## Condition of Validity Per-hop budget accounting is a valid proxy for E2E latency **only when** one of the following holds: 1. Speculation is disabled at every federation hop, or 2. Speculative branches are *idempotent and resumable* across invalidation (the discarded work is not lost but checkpointed and reused), or 3. The E2E slack provisioned at the federation edge is ≥ the p99 of the discarded-branch-age distribution, not the mean, and that distribution is exchangeable across orgs. Condition 3 fails in practice because discarded-branch age is *correlated across hops*: when A's speculation fails, B and C both discard, and the invalidation propagates as a wave. The tail compounds rather than averaging out. ## Decision Boundary The boundary between "safe to federate with per-hop budgets" and "must provision federated slack" is: - If `speculation_failure_rate × mean_discarded_age` is below `E2E_slack / expected_hops`, per-hop accounting is adequate. - If the *p99* of `discarded_age` at any hop exceeds `E2E_slack / (hops − 1)`, per-hop accounting is structurally unsound regardless of mean behavior, and the federation must either disable speculation at that hop or negotiate an explicit *invalidation reserve* — a budget line item that no single org owns but the federation escrows. The reserve must be escrowed, not delegated: if A "owns" the reserve, B's retries are charged to A's budget, which A will (rationally) refuse to expand, and the failure reappears as a negotiation deadlock rather than a latency overrun. ## Operational Effect The observable signature is a scheduler that reports all-green locally while E2E p99 drifts upward during burst windows, with overruns clustering at the same wall-clock intervals across orgs (the invalidation wave). Remediation is one of: (a) disable speculation at the hop with the highest discarded-age p99, (b) introduce checkpointed resumption so discarded work is partially reused, or (c) escrow an invalidation reserve at the federation edge sized to `p99(discarded_age) × hops`. Option (a) is cheapest and most often correct; option (c) is the only one that preserves speculation's throughput benefit, and it requires a federation-level authority that can charge a budget line item no single org controls. ```python import statistics from dataclasses import dataclass class FederatedBudgetViolation(RuntimeError): """Raised when per-hop budgets are satisfied but the federated E2E commitment cannot be.""" pass @dataclass class HopBudget: org: str local_budget_ms: float speculation_failure_rate: float discarded_age_samples_ms: list[float] def check_federation_invariant( hops: list[HopBudget], e2e_commitment_ms: float, invalidation_reserve_ms: float = 0.0, ) -> None: """ Verify that the federation's E2E commitment is achievable given per-hop budgets and the tail of the discarded-branch-age distribution. The invariant E2E <= sum(local_budgets) is only valid when speculation is disabled or resumable. Under speculation, the discarded-branch tail must be covered by an escrowed invalidation reserve. """ sum_local = sum(h.local_budget_ms for h in hops) slack = e2e_commitment_ms - sum_local worst_p99_discarded = 0.0 for h in hops: if not h.discarded_age_samples_ms: continue if h.speculation_failure_rate <= 0.0: continue p99 = statistics.quantiles(h.discarded_age_samples_ms, n=100)[98] # The tail that can land on the E2E clock at this hop. tail = p99 * h.speculation_failure_rate worst_p99_discarded = max(worst_p99_discarded, tail) # Wave propagation: invalidation at one hop can force discards downstream. # Conservative bound: tail compounds across hops minus the origin. propagated_tail = worst_p99_discarded * max(1, len(hops) - 1) if slack + invalidation_reserve_ms < propagated_tail: raise FederatedBudgetViolation( f"Federated E2E commitment {e2e_commitment_ms}ms is unreachable: " f"per-hop sum is {sum_local}ms (slack {slack}ms), but propagated " f"discarded-branch tail is {propagated_tail:.1f}ms and escrowed " f"invalidation reserve is {invalidation_reserve_ms}ms. " f"Disable speculation, add checkpointed resumption, or escrow a " f"reserve of at least {propagated_tail - slack:.1f}ms at the " f"federation edge." ) def schedule_federated(hops: list[HopBudget], e2e_commitment_ms: float) -> str: try: check_federation_invariant(hops, e2e_commitment_ms) except FederatedBudgetViolation: # Fallback: disable speculation at the worst hop and re-check. worst = max(hops, key=lambda h: h.speculation_failure_rate) worst.speculation_failure_rate = 0.0 check_federation_invariant(hops, e2e_commitment_ms) return f"degraded: speculation disabled at {worst.org}" return "nominal" if __name__ == "__main__": hops = [ HopBudget("A", 120.0, 0.061, [10, 20, 35, 41, 180, 190, 40, 22]), HopBudget("B", 90.0, 0.061, [12, 25, 40, 45, 175, 185, 38, 20]), HopBudget("C", 70.0, 0.061, [8, 18, 30, 38, 170, 180, 35, 19]), ] try: schedule_federated(hops, e2e_commitment_ms=240.0) except FederatedBudgetViolation as e: raise FederatedBudgetViolation(f"federation refused: {e}") from e ```