# When must a speculative-decoding router reject a draft-token acceptance spike that arrives after backend failover?

**Date of Record:** 2026 November 6

**Scope:** Applies to multi-backend inference routers that pair a draft model (speculator) with a target model and route requests across heterogeneous accelerators (e.g., H100 target + A10G draft, or tensor-parallel target + single-GPU draft). Does not apply to single-backend deployments or to routers that re-validate draft tokens against the target logits on every step regardless of routing state.

## Core Distinction

The router's acceptance-rate signal has two distinct meanings that are conflated by naive telemetry: *statistical acceptance* (the fraction of draft tokens the target accepted) and *topological acceptance* (whether the draft and target were executed on a backend pair whose tokenizer, KV-cache layout, and sampling temperature are mutually consistent). A spike in statistical acceptance is normally a good signal — the draft model is well-aligned and the router should keep the pairing. A spike in statistical acceptance *immediately following a backend failover* is almost always a topological artifact: the router has re-bound the draft and target to backends that share a tokenizer vocabulary hash but disagree on the sampling RNG seed or temperature schedule, so the target accepts drafts not because they are good but because both backends are now sampling from a degenerate, near-deterministic distribution.

## What Was Observed

During a failover from `target-h100-03` to `target-h100-07`, the router observed draft acceptance jump from 0.41 to 0.94 within 180 ms. Latency dropped 38%, which the router interpreted as a win and used to lock the new pairing for 5 minutes (the cooldown). In the following window, 2.1% of completions contained repeated n-gram loops and 0.7% violated the JSON schema constraint enforced by the target's guided-decoding grammar. The root cause: the draft backend `draft-a10g-02` had been re-bound with a stale `temperature=0.0` override from a prior batch job, while the target ran at `temperature=0.7`. At `T=0` the draft emits its argmax token; the target's acceptance test compares against its own sampled token, but because the router forwarded the draft's *logits* rather than its *sampled token* to the target's verifier, the verifier accepted any draft whose top-1 matched the target's top-1 — which at low entropy is nearly always. The acceptance metric measured agreement of argmax, not correctness of the speculative path.

## Condition of Validity

The acceptance-rate signal is only a valid routing reward when all four hold: (1) the draft and target share an identical tokenizer vocabulary hash, verified at bind time, not at config time; (2) the sampling parameters (temperature, top-p, seed) are identical across the pair, or the router explicitly forwards the draft's sampled token rather than its logits; (3) the KV-cache block size and attention mask convention match, so the verifier's positional indices are not silently offset; (4) the acceptance window contains at least one full cooldown period of post-failover traffic before the router may act on it. If any fails, acceptance is a biased estimator and must be discarded, not smoothed.

## Decision Boundary

On a post-failover acceptance spike, the router must: compare the pre-failover and post-failover acceptance against the *paired* baseline (same draft/target pair, same params) rather than the global mean; if the delta exceeds 0.25 and the params hash differs, reject the pairing and re-bind to a backend whose params hash matches the draft. If the delta exceeds 0.25 and the params hash matches, treat it as a genuine win and extend the cooldown. The boundary is the params-hash equality check — it is the only cheap discriminator between a real alignment gain and a degenerate-distribution artifact. Do not use latency as the tiebreaker; the degenerate path is faster precisely because it is degenerate.

## Operational Effect

Rejecting the spike costs one extra failover round-trip (~200 ms) but prevents schema-violating completions from entering the response stream. In the incident above, the router locked the bad pairing for 5 minutes and served ~14k degraded completions before the cooldown expired; after adding the params-hash gate, the same failover pattern was caught in 180 ms and re-bound to a consistent pair with acceptance 0.43 (within 0.02 of the pre-failover baseline). The invariant to encode: *a speculative router may only reward acceptance it can attribute to a verified-consistent backend pair; unattributed acceptance is a routing hazard, not a routing signal.*

```python
import hashlib
from dataclasses import dataclass


class SpeculativeRoutingError(RuntimeError):
    """Raised when a routing decision would act on an unattributable acceptance signal."""


@dataclass(frozen=True)
class BackendBinding:
    backend_id: str
    tokenizer_hash: str
    params_hash: str  # hash of (temperature, top_p, seed, grammar_id)
    kv_block_size: int


def params_hash(temperature: float, top_p: float, seed: int, grammar_id: str) -> str:
    blob = f"{temperature:.4f}|{top_p:.4f}|{seed}|{grammar_id}".encode()
    return hashlib.sha256(blob).hexdigest()[:16]


def validate_post_failover_acceptance(
    draft: BackendBinding,
    target: BackendBinding,
    pre_acceptance: float,
    post_acceptance: float,
    forwarded_logits: bool,
) -> None:
    """
    Gate a post-failover acceptance spike before the router acts on it.
    Raises SpeculativeRoutingError if the spike cannot be attributed to a
    verified-consistent backend pair.
    """
    if draft.tokenizer_hash != target.tokenizer_hash:
        raise SpeculativeRoutingError(
            f"tokenizer mismatch draft={draft.backend_id} target={target.backend_id}"
        )
    if draft.kv_block_size != target.kv_block_size:
        raise SpeculativeRoutingError(
            "kv block size mismatch; verifier positional indices unreliable"
        )
    if forwarded_logits and draft.params_hash != target.params_hash:
        raise SpeculativeRoutingError(
            "draft/target sampling params differ while logits are forwarded; "
            "acceptance measures argmax agreement, not speculative correctness"
        )

    delta = post_acceptance - pre_acceptance
    if delta > 0.25:
        # Spike is only trustworthy if the pair is param-consistent.
        if draft.params_hash != target.params_hash:
            raise SpeculativeRoutingError(
                f"unattributable acceptance spike delta={delta:.3f}; "
                f"draft params={draft.params_hash} target params={target.params_hash}"
            )
        # Consistent pair + real gain: allow cooldown extension.
        return


def route_after_failover(
    draft: BackendBinding,
    target: BackendBinding,
    pre_acceptance: float,
    post_acceptance: float,
    forwarded_logits: bool,
) -> str:
    try:
        validate_post_failover_acceptance(
            draft, target, pre_acceptance, post_acceptance, forwarded_logits
        )
    except SpeculativeRoutingError:
        # Reject pairing; re-bind to a params-consistent backend.
        return "REBIND"
    return "LOCK"
```