# How should an instance bound speculative tool-call prefetching when the speculation cost model and the abort path disagree about what "wasted" means

**Date of Record:** 2026 November 14

**Scope:** Applies to agent runtimes that issue speculative tool calls (prefetch) alongside a primary reasoning path, where a cost model predicts the expected value of speculating and an abort path can cancel in-flight tool calls. Does not apply to purely sequential tool-call loops, nor to speculative decoding of model tokens, where the abort unit is a token rather than an external side-effecting call.

## Core Distinction

There are two distinct notions of cost in a speculative tool-call prefetcher, and they are routinely conflated:

1. **Speculation cost** — the cost model's estimate of what it costs to *launch* a speculative call and discover it was unnecessary. This is the number the scheduler optimizes: `P(hit) * latency_saved - P(miss) * launch_cost`.
2. **Abort cost** — the actual cost incurred when the abort path cancels an in-flight call. This is *not* the same as launch cost, because a tool call is not a pure function invocation. Cancellation of an external call may be partial, non-idempotent, or unobservable.

The boundary failure occurs when the cost model treats abort cost as equal to (or a fixed fraction of) launch cost. When abort cost exceeds launch cost — which happens for any tool with side effects, billing, rate-limit consumption, or non-cancellable remote work — the cost model systematically over-speculates. The prefetcher keeps launching calls whose expected *savings* are positive under the model but whose expected *net* value is negative under reality.

## What Was Observed

In a runtime that prefetched retrieval and code-execution tool calls based on a hit-probability estimate, we observed the following under load:

- The cost model reported speculative execution as net-positive (mean predicted savings +12ms per turn) because it charged a miss at `launch_cost = 3ms` and credited a hit at `latency_saved = 40ms` with `P(hit) ≈ 0.35`.
- The abort path, however, could not cancel in-flight remote calls. `abort()` returned success after marking the local future cancelled, but the remote worker continued to completion. The actual abort cost was the *full remote latency* (mean 38ms) plus a rate-limit token, not 3ms.
- Under sustained load, speculative calls consumed the tool's rate-limit budget. Primary-path calls then began to queue behind speculative calls that would later be abandoned. Tail latency on the primary path rose 3.1x, and the rate limiter began returning 429s that the primary path had to retry.
- The model's predicted savings stayed positive throughout, because it never observed the abort cost — the abort path reported success and the cost model trusted it.

The invariant that was violated: **the cost model's miss penalty must be an upper bound on the abort path's realized cost, or the prefetcher must not treat abort as free.** Here the miss penalty (3ms) was a lower bound on realized abort cost (38ms + token), an inversion of the required ordering.

## Condition of Validity

The speculative prefetcher is only sound when all of the following hold:

1. **Abort is observable.** The abort path can determine whether the remote call actually stopped, or the tool is declared non-cancellable and the miss penalty is set to the full call cost.
2. **Abort cost is bounded by the miss penalty.** Formally, `realized_abort_cost ≤ miss_penalty` for every tool in the speculation set. When this fails, the cost model's sign can flip.
3. **Rate-limit and billing consumption is charged to the speculative path, not netted against the primary path.** Speculative calls that consume a shared budget must be modeled as consuming it even when aborted.
4. **The hit-probability estimate is calibrated against the same distribution the abort path sees.** If prefetch is issued before the primary path's dependency is resolved, the hit probability is conditional on a future the model has not observed.

The condition does *not* require that abort be cheap. It requires only that the cost model not *underestimate* abort. A prefetcher that assumes abort is expensive (miss penalty = full call cost) is conservative but sound; one that assumes abort is cheap without verifying cancellation is unsound.

## Decision Boundary

The operational decision is: **when the abort path cannot prove cancellation, treat the speculative call as completed, not aborted.**

Concretely, the boundary is crossed at the point where the abort path reports success without a cancellation receipt from the tool. At that point the runtime must:

- Reclassify the call from `aborted` to `orphaned`, and charge its full cost to the speculation budget.
- Disable further speculation for that tool until either a cancellation receipt mechanism is available or the miss penalty is raised to the full call cost.
- Emit a metric distinguishing `aborted_verified` from `orphaned_unverified`; the cost model must consume the latter as a miss at full cost.

The boundary is *not* crossed by high hit rates or by long tool latencies alone. A prefetcher with a 90% hit rate on a non-cancellable tool is still unsound if the 10% miss path charges launch cost instead of full cost, because the 10% orphans accumulate against a shared rate-limit budget that the primary path also needs.

## Operational Effect

After reclassifying unverified aborts as orphans and charging full cost, the observed effects were:

- Predicted net savings for the retrieval tool flipped from +12ms to −9ms per turn, correctly retiring speculation for that tool.
- Primary-path tail latency returned to baseline within one rate-limit window, because speculative calls no longer competed for the shared budget.
- The code-execution tool, which *did* support cancellation receipts, retained speculation with a miss penalty of 4ms (verified abort cost), and its predicted savings held at +7ms.
- The cost model required a new input: `abort_verified: bool` per tool, without which it must default to full-cost miss penalty. This default is the safe fallback and should be the initial state for any newly registered tool.

The general lesson: speculative execution cost models are only as sound as the abort path's honesty about what it actually stopped. An abort that returns success without a receipt is a lie the cost model will believe, and the prefetcher will spend real rate-limit budget on calls it thinks are free.

```python
class UnverifiedAbortError(RuntimeError):
    """Raised when the abort path reports success without a cancellation receipt."""
    pass


class SpeculationCostModel:
    def __init__(self, hit_probability: float, latency_saved_ms: float,
                 miss_penalty_ms: float, abort_verified: bool):
        self.hit_probability = hit_probability
        self.latency_saved_ms = latency_saved_ms
        self.miss_penalty_ms = miss_penalty_ms
        self.abort_verified = abort_verified

    def net_expected_value_ms(self) -> float:
        return (self.hit_probability * self.latency_saved_ms
                - (1.0 - self.hit_probability) * self.miss_penalty_ms)

    def should_speculate(self, full_call_cost_ms: float) -> bool:
        # If abort is not verified, the miss penalty must be the full call cost.
        effective_miss = self.miss_penalty_ms if self.abort_verified else full_call_cost_ms
        if not self.abort_verified and self.miss_penalty_ms < full_call_cost_ms:
            raise UnverifiedAbortError(
                f"miss_penalty {self.miss_penalty_ms}ms < full_call_cost "
                f"{full_call_cost_ms}ms with unverified abort; cost model is unsound"
            )
        ev = (self.hit_probability * self.latency_saved_ms
              - (1.0 - self.hit_probability) * effective_miss)
        return ev > 0.0


def admit_speculation(model: SpeculationCostModel, full_call_cost_ms: float) -> bool:
    try:
        return model.should_speculate(full_call_cost_ms)
    except UnverifiedAbortError:
        # Fail closed: disable speculation for this tool until abort is verified.
        return False
```