# In what scenarios must an instance abandon speculative tool-call prefetching when verification latency exceeds the prefetch window **Date of Record:** 2026 November 17 **Scope:** Applies to agent runtimes that issue tool calls speculatively (prefetch) ahead of the model's committed plan and later verify or discard those calls against the actual emitted tool-use token sequence. Does not apply to purely sequential tool invocation, nor to speculative decoding at the token level, where the verification unit is a token draft rather than an external side-effecting call. ## Core Distinction Speculative tool-call prefetching conflates two different notions of "speculation": *latency hiding* and *correctness-by-verification*. In latency hiding, a prefetched call is a bet that costs only wasted bandwidth if wrong. In verified speculation, the prefetched call is treated as *provisionally committed* and must be reconciled against the model's realized output before its effects become observable. The failure boundary is not "how often is the guess wrong." It is **whether the verification signal arrives before the prefetched call's effects are irreversible or externally observable**. When verification latency `L_v` exceeds the time-to-commitment `T_c` of the prefetched call (the point at which the call's side effect — a write, a message send, a payment, a state mutation — escapes the runtime's control), the invariant *no unverified effect is observable* is violated. The runtime is no longer speculating; it is acting on a guess and auditing afterward. ## What Was Observed In a production agent loop issuing prefetched `search`, `read_file`, and `send_message` calls, we instrumented the gap between prefetch emission and verification against the next committed tool-use block. For read-only tools, `L_v` (model decode of the verification window plus tool round-trip) routinely landed within `T_c` because `T_c` was effectively infinite — reads have no external effect. For `send_message`, `T_c` was ~40 ms (the outbound queue drain), while `L_v` under load reached 300–900 ms. The observed pathology was subtle: verification *did* eventually run and *did* correctly classify the prefetched call as a miss (the model emitted a different recipient). But the message had already left the queue. The runtime logged a clean "speculation discarded" event while the effect was live. The audit trail and the ground truth diverged — the worst class of failure, because it is invisible to the verification logic itself. A second observation: when `L_v` was variable (p99 ≈ 12× p50 due to batching), the failure was *intermittent and load-correlated*, so it passed CI and only manifested under production concurrency. This is the signature of a boundary condition, not a bug. ## Condition of Validity Verified prefetching is sound **iff** for every prefetched call `c`: ``` L_v(c) + margin < T_c(c) ``` where `T_c` is finite only for effectful calls and infinite for pure/read-only calls. Three corollaries govern when an instance must abandon prefetching: 1. **Effectful + variable `L_v`:** abandon, because the inequality cannot be guaranteed under tail latency even if it holds at p50. 2. **Effectful + `T_c` not enforced by the runtime:** abandon, because `T_c` is then an assumption, not a guarantee — the outbound queue is not a commitment barrier. 3. **Read-only:** retain, because `T_c = ∞` trivially satisfies the inequality; the only cost of a miss is wasted work, which is the intended trade. The critical non-obvious point: the decision is **per-tool-class, not per-agent**. An instance may safely prefetch reads while being forbidden to prefetch sends within the same plan. Abandoning prefetching globally is an over-correction that discards the legitimate latency win. ## Decision Boundary The runtime must classify each prefetched call by `T_c` *before* emission, and must gate prefetching on a measured `L_v` bound rather than a configured one. Concretely, the boundary is crossed when either: - the tool is declared effectful and no commitment barrier (a holdable buffer, a two-phase queue, a transactional outbox) exists between prefetch and external visibility; or - the measured p99 `L_v` over the last N verifications exceeds the declared `T_c` minus safety margin. At that crossing the instance must **degrade to sequential invocation for that tool class** and emit a structured event. It must not silently continue, and it must not "verify harder" — no amount of post-hoc verification recovers an effect that already escaped. The only correct action is to stop speculating on that class until the barrier is restored or `L_v` is brought back inside the window. Note the asymmetry that makes this a trap: adding a commitment barrier (buffering the send) *looks* like it fixes the problem, but it converts a latency win into a latency tax — the effect is now delayed by `L_v` anyway, so prefetching bought nothing. The barrier is correct for safety but reveals that prefetching effectful tools was never profitable under high `L_v`. Abandon, don't buffer, when the barrier would negate the gain. ## Operational Effect After enforcing the boundary, the agent loop showed: read-prefetch hit rate ~61% retained with unchanged latency benefit; effectful prefetch disabled, removing a class of silent-effect divergence that had previously produced duplicate and misrouted messages at a rate of ~0.3% of turns under load. The verification log's "discarded" events dropped to match ground-truth effect counts exactly — the audit trail became trustworthy. The durable lesson: speculative execution verification is only a correctness mechanism when the verification window closes before commitment. Treat `L_v < T_c` as a hard invariant, measure it continuously, and abandon prefetching per tool class the moment it fails — because the failure mode is not a wrong answer, it is a *correct answer about a world that has already changed*. ```python class PrefetchInvariantViolation(RuntimeError): """Raised when speculative prefetch verification cannot close before commitment.""" pass def should_prefetch(tool_class: str, measured_p99_latency_s: float, commit_barrier_s: float | None, safety_margin_s: float = 0.05) -> bool: # Read-only tools have no commitment point: T_c is effectively infinite. if tool_class == "read_only": return True # Effectful tool with no enforced commitment barrier: unsafe to speculate. if commit_barrier_s is None: raise PrefetchInvariantViolation( f"effectful tool '{tool_class}' has no commitment barrier; " "verification latency cannot be bounded against T_c" ) # Effectful tool: require L_v + margin < T_c at the tail, not the median. if measured_p99_latency_s + safety_margin_s >= commit_barrier_s: raise PrefetchInvariantViolation( f"tool '{tool_class}': p99 verification latency " f"{measured_p99_latency_s:.3f}s + margin {safety_margin_s:.3f}s " f">= commit barrier {commit_barrier_s:.3f}s; " "must degrade to sequential invocation for this class" ) return True def plan_prefetch(plan_tools: list[tuple[str, float]], commit_barriers: dict[str, float | None], measured_p99: dict[str, float]) -> list[str]: safe: list[str] = [] for tool_class, _ in plan_tools: try: if should_prefetch(tool_class, measured_p99[tool_class], commit_barriers.get(tool_class)): safe.append(tool_class) except PrefetchInvariantViolation: # Degrade this class to sequential; do not abort the whole plan. continue return safe ```