# Where must an instance refuse to speculate: branch-and-verify rollback under cost-aware fan-out **Date of Record:** 2026 November 18 **Scope:** Applies to orchestrators that fan out multiple speculative agent branches, verify each candidate, and roll back non-verified branches while accounting for per-token and per-call cost. Does not apply to single-agent retry loops, deterministic workflow engines, or systems where branch verification is free and instantaneous. ## Core Distinction The common mental model treats speculative fan-out as a latency/cost tradeoff: spawn N branches, verify, keep the winner, discard the losers. The invariant that actually governs correctness is different. **Rollback is only sound if the side-effect boundary of every branch is strictly below the verification boundary.** An instance must refuse to speculate when a branch's *observable* effects can escape before verification completes — because "roll back" then becomes "compensate," and compensation is not rollback. Two failure surfaces are conflated in practice: 1. **Cost rollback** — the accounting of tokens/calls already spent. This is always recoverable; you simply record it. 2. **Effect rollback** — the reversal of world-state changes (writes, sends, external API mutations). This is recoverable only if the effect was never externally visible. Speculation is safe only when effect rollback is guaranteed by construction. When it is not, the branch must be *verified before it acts*, not after. ## What Was Observed A multi-agent planner fanned out three speculative branches to fulfill a request. Each branch was a full agent with tool access. Branch A produced a correct plan and executed a `send_email` tool call; branches B and C produced plausible but ultimately rejected plans. The verifier ran *after* branch completion, selected A, and rolled back B and C — including refunding their token budgets to the cost ledger. The cost ledger was consistent. The world state was not. Branch B had already issued a `create_calendar_event` call to an external provider during its reasoning, before the verifier ever saw it. The rollback routine, which only knew about ledger entries and in-process state, reported a clean rollback. The external calendar now held a phantom event. No error was raised at any layer, because every layer's local invariant held: the ledger balanced, the in-process branch state was discarded, and the verifier had selected a valid winner. The non-obvious part: the bug was not in the verifier, the rollback, or the cost accounting. It was in the *fan-out admission decision* — the system speculated a branch whose effect boundary was above its verification boundary. ## Condition of Validity Speculative fan-out with rollback is valid if and only if, for every branch: ``` effect_boundary(branch) < verification_boundary(branch) ``` Concretely, a branch is speculatable only if all of the following hold: - Every tool call it may issue is either **read-only** or **deferred** (queued, not dispatched) until the branch is selected. - The verifier's decision is made on a *proposal* artifact, not on a *committed* effect. - Any external mutation is gated behind a commit token that is issued only after verification returns `SELECTED`. - Cost accounting is separated from effect accounting, so a "clean rollback" of the ledger cannot be mistaken for a clean rollback of the world. If any branch can reach an externally observable mutation before verification, the invariant fails and the instance must not speculate that branch. It may still fan out *pure* branches (planning, retrieval, drafting) and serialize only the effecting step. ## Decision Boundary The admission rule is a single predicate evaluated at fan-out time, before any branch is spawned: - **Speculate** when the branch's declared tool set is read-only or deferrable, and the verifier can decide on the proposal alone. - **Serialize** when the branch requires an externally visible mutation whose result the verifier depends on. Here, verification must precede effect: run one branch, verify, then act. - **Refuse** when the branch's tool set is unknown or under-declared. Unknown effect boundaries are treated as unsafe — the instance must not speculate, because it cannot prove the ordering invariant. The cost-aware dimension sharpens this: fan-out is only worth paying for on branches whose effects are cheap to discard. A branch that is expensive *and* effectful is the worst case — you pay the cost, you cannot roll back the effect, and you cannot amortize by keeping the branch. ## Operational Effect Enforcing the predicate changes the failure mode from silent world-state corruption to an explicit, catchable refusal at admission time. The orchestrator no longer reports "clean rollback" for branches that leaked effects, because such branches are never admitted to speculation. The cost is reduced parallelism on effectful steps; the benefit is that rollback means rollback. The following guard encodes the admission predicate and raises when an instance is asked to speculate a branch whose effect boundary is not provably below its verification boundary. ```python class UnsafeSpeculationError(RuntimeError): """Raised when a branch is admitted to speculative fan-out but its effect boundary is not provably below its verification boundary.""" def admit_branch(branch, verifier, *, deferrable_tools: set[str]) -> None: """Gate a branch into speculative fan-out. Speculation is sound only if every tool the branch may invoke is either read-only or deferred until after verification. Unknown or effectful tools must be serialized, never speculated. """ if not getattr(branch, "tools_declared", False): raise UnsafeSpeculationError( f"branch {branch.id!r} has undeclared tools; effect boundary unknown" ) for tool in branch.tools: if tool.is_read_only: continue if tool.name in deferrable_tools and tool.supports_deferral: continue raise UnsafeSpeculationError( f"branch {branch.id!r} may invoke effectful tool {tool.name!r} " f"before verification; serialize instead of speculating" ) if not verifier.decides_on_proposal_only: raise UnsafeSpeculationError( f"verifier for branch {branch.id!r} inspects committed effects; " f"verification boundary is not above effect boundary" ) ```