# How should an instance reconcile speculative draft tokens with parallel tool-call side effects when the verifier accepts a token after a sibling branch has already committed an irreversible action

**Date of Record:** 2026 November 14

**Scope:** Applies to agentic inference stacks that combine speculative decoding (draft model proposes N tokens, target model verifies) with parallel tool-call orchestration where multiple tool invocations are dispatched speculatively and may produce non-idempotent side effects. Does not apply to pure text generation without tool dispatch, nor to orchestration layers where all tool calls are transactional or committed only after a single verification barrier.

## Core Distinction

The failure mode is not a decoding error and not a tool error. It is a *cross-layer consistency violation* between two independent optimistic mechanisms that were each correct in isolation.

Speculative decoding is optimistic at the token level: a draft model emits a candidate suffix, the verifier accepts a prefix of it and rejects the remainder. The invariant it relies on is that *rejected tokens produce no observable effect* — they are simply discarded and re-decoded. Parallel tool-call orchestration is optimistic at the action level: the orchestrator dispatches sibling tool calls concurrently under the assumption that their effects are commutable or that a later reconciliation pass will resolve conflicts.

When these mechanisms are stacked, the draft model can propose a tool-call token sequence, the orchestrator can dispatch the corresponding side effect *before the verifier has ruled on the tokens that produced it*, and the verifier can then reject those tokens. The rejected tokens are discarded — but the side effect they caused is not. The speculative-decoding invariant ("rejected tokens are inert") and the orchestration invariant ("sibling effects are reconcilable") are each locally true and jointly false.

The non-obvious part: the violation is *silent*. The token stream is valid, the tool log is valid, the final answer may even be correct. The corruption lives only in the gap between what the model "said" (post-verification token stream) and what the world did (pre-verification side effects).

## What Was Observed

A three-branch agent turn: branch A issued a `CREATE` on a resource, branch B issued a `DELETE` on a sibling resource, branch C issued a `READ` used to compose the final response. Draft model emitted all three tool-call blocks in one speculative window of 48 tokens. The orchestrator, optimizing for latency, dispatched A and B as soon as their closing delimiters appeared in the *draft* stream — 11 tokens before the verifier reached them.

The verifier rejected 9 tokens spanning the argument region of branch A (a resource name was malformed). Under normal speculative decoding this is a benign event: re-decode from the last accepted token. But branch A's `CREATE` had already executed against the malformed name. The orchestrator's reconciliation pass, keyed on the *accepted* token stream, saw branch A as never having been issued (its call block was inside the rejected span) and therefore never issued a compensating `DELETE`. The malformed resource persisted.

Meanwhile branch B's `DELETE` was accepted and executed. Branch C's `READ` had been dispatched against the pre-`DELETE` state and returned a stale value that the final response cited. The response was internally consistent with the accepted token stream and externally inconsistent with the world.

Three distinct anomalies, one root cause: side effects were committed against draft tokens whose acceptance was still undecided, and the reconciliation key was the accepted stream rather than the dispatched stream.

## Condition of Validity

The speculative-decoding inertness invariant holds **if and only if** every effect attributable to a token is gated on that token's acceptance. Formally: for any token `t` in a draft window, `effect(t)` must not be observable until `accept(t)` is decided, and if `¬accept(t)` then `effect(t) = ∅`.

Parallel orchestration breaks this in two ways:

1. **Eager dispatch.** The orchestrator treats the draft stream as a commit log. This is the obvious violation and is easy to catch with a review.
2. **Commit-keyed reconciliation.** Even a lazy orchestrator that waits for acceptance can violate the invariant if its reconciliation logic keys on the accepted stream while its dispatch log keys on the draft stream. The two keys diverge exactly on rejected spans, and the divergence is invisible unless you diff them.

The condition of validity therefore requires a *single commit key* shared by dispatch, reconciliation, and the accepted token stream — and dispatch must be deferred until the verifier's decision for the enclosing call block is final. Partial acceptance within a call block (verifier accepts the opening delimiter but rejects an argument token) must be treated as whole-block rejection for dispatch purposes; there is no safe partial commit of a tool call.

## Decision Boundary

The operator faces a genuine tradeoff, not a bug to be patched away:

- **Defer dispatch to verification.** Restores the invariant. Cost: you lose the latency win that motivated speculative tool dispatch in the first place — tool round-trips now serialize behind the verifier, and the verifier's own latency is on the critical path.
- **Dispatch eagerly, reconcile on the union key.** Preserves latency. Cost: you must implement compensation for every rejected-span side effect, which requires every tool to be either idempotent, reversible, or accompanied by an explicit compensating action. Most real tool surfaces are none of these.
- **Restrict speculation to read-only tools.** Dispatch reads eagerly, defer writes to verification. This is the boundary most production systems converge on, and it is the correct default. The boundary condition is: *speculation is safe exactly for effects that are observationally inert under rejection* — reads qualify, writes do not, and "idempotent write" qualifies only if the idempotency key is derived from the accepted token stream, which reintroduces the key-divergence problem.

The decision boundary is not "how much latency can we tolerate" but "which effects are safe to commit under uncertainty." That question is answerable per-tool, not per-turn, and the answer must be enforced at dispatch time rather than audited afterward.

## Operational Effect

The operational consequence is that speculative decoding and parallel orchestration cannot be composed as independent optimizations. Each assumes the other is absent: speculative decoding assumes effects are inert until acceptance, orchestration assumes effects are reconcilable after the fact. Stacked, they produce a class of corruption that is invisible to both layers' own invariants and to end-to-end correctness checks that compare the response to the accepted token stream.

Mitigation requires an explicit effect-classification table consulted at dispatch, a single commit key threaded through dispatch, reconciliation, and the accepted stream, and a hard rule that no tool call block is dispatched until the verifier's decision for that block is final. The compensating-action fallback is a trap: it works only for reversible tools, and the tools where you most want speculative dispatch (long-latency writes) are exactly the ones least likely to be reversible.

```python
from dataclasses import dataclass, field
from enum import Enum
from typing import Callable


class SpeculativeEffectViolation(RuntimeError):
    """Raised when a side effect is committed against undecided draft tokens."""
    pass


class EffectClass(Enum):
    READ_ONLY = "read_only"      # inert under rejection: safe to dispatch eagerly
    IDEMPOTENT = "idempotent"    # safe only if key derives from accepted stream
    WRITE = "write"              # must defer to verification


@dataclass
class DraftToken:
    index: int
    text: str
    accepted: bool | None = None  # None = undecided


@dataclass
class ToolCallBlock:
    name: str
    effect: EffectClass
    token_span: tuple[int, int]  # [start, end) in the draft stream
    dispatch: Callable[[], object]
    idempotency_key: str | None = None


@dataclass
class Orchestrator:
    commit_key: str
    dispatched: list[tuple[str, str]] = field(default_factory=list)  # (call_name, commit_key)

    def dispatch_block(self, block: ToolCallBlock, tokens: list[DraftToken]) -> object | None:
        span = tokens[block.token_span[0]:block.token_span[1]]

        # The invariant: no effect may be committed against an undecided token.
        if any(t.accepted is None for t in span):
            if block.effect is EffectClass.READ_ONLY:
                # Reads are inert under rejection; dispatching early is safe.
                self.dispatched.append((block.name, self.commit_key))
                return block.dispatch()
            raise SpeculativeEffectViolation(
                f"tool {block.name!r} (effect={block.effect.value}) "
                f"dispatched against undecided tokens in span "
                f"{block.token_span}; commit_key={self.commit_key!r}. "
                f"Defer dispatch until the verifier rules on the enclosing block."
            )

        if all(t.accepted for t in span):
            # Whole block accepted: safe to commit under the shared key.
            if block.effect is EffectClass.IDEMPOTENT and block.idempotency_key != self.commit_key:
                raise SpeculativeEffectViolation(
                    f"idempotency key {block.idempotency_key!r} diverges from "
                    f"commit key {self.commit_key!r}; reconciliation will miss this effect."
                )
            self.dispatched.append((block.name, self.commit_key))
            return block.dispatch()

        # Block contains a rejected token: no effect, no dispatch, no reconciliation entry.
        return None


def reconcile(orch: Orchestrator, accepted_stream_key: str) -> list[str]:
    """Return dispatch entries orphaned by key divergence."""
    if accepted_stream_key != orch.commit_key:
        raise SpeculativeEffectViolation(
            f"reconciliation keyed on accepted stream {accepted_stream_key!r} but "
            f"dispatch log keyed on {orch.commit_key!r}; rejected-span effects "
            f"({len(orch.dispatched)} entries) are invisible to this pass."
        )
    return [name for name, _ in orch.dispatched]
```