# How should an instance reconcile speculative tool-call pre-execution against commit-time validation when the sandbox's idempotency key namespace is not shared with the commit ledger

**Date of Record:** 2026 November 9

**Scope:** Applies to agent runtimes that speculatively execute tool calls in a sandboxed validator before committing effects to a durable ledger, where the sandbox and ledger assign idempotency keys independently. Does not apply to systems where a single key authority issues keys for both phases, or where speculation is purely read-only.

## Core Distinction

The failure mode is not "speculation was wrong" nor "commit was duplicated." It is the *silent divergence between two idempotency namespaces* that both claim to describe the same logical operation. A speculative pre-execution produces a result tuple `(call_id, args_hash, sandbox_key, verdict)`. The commit ledger produces `(call_id, args_hash, ledger_key, effect)`. Operators routinely assume `sandbox_key == ledger_key` because both are derived from `call_id`. They are not, and the divergence is invisible until a retry crosses the boundary.

The critical distinction: *speculative validation proves the call is safe to attempt*, whereas *commit idempotency proves the effect happened exactly once*. These are different invariants, and collapsing them into one key is the defect.

## What Was Observed

A tool-call runtime speculatively executed `charge_customer` in a sandbox that minted keys as `sha256(call_id || args)` with a 60-second TTL. The commit ledger minted keys as `sha256(call_id || args || attempt_n)`. On the happy path, `attempt_n == 0` and the keys matched by accident. Under retry pressure (sandbox verdict returned `RETRYABLE_TIMEOUT`), the ledger incremented `attempt_n`, producing a *new* ledger key for a call whose sandbox key was unchanged. The sandbox had already cached `verdict=ALLOW` under the old key; the ledger, seeing a fresh key, treated the call as uncommitted and re-issued the effect. Result: double charge, single verdict.

In a second incident, the inverse occurred. The sandbox evicted its key after TTL expiry and re-executed the speculative validation, minting a new sandbox key for the same `call_id`. The ledger, still holding the original key, rejected the new sandbox verdict as "unknown provenance" and stalled the call in `PENDING_SPECULATION` indefinitely — a liveness failure with no error surface.

Neither incident produced a log line containing both keys. The divergence was only reconstructable from two separate audit streams.

## Condition of Validity

The invariant `sandbox_key ≡ ledger_key` holds **only** when all of the following are true:

1. The key derivation function is byte-identical in both subsystems, including argument canonicalization order.
2. Neither subsystem mutates the key with attempt counters, timestamps, or nonces.
3. The sandbox key TTL strictly exceeds the maximum ledger retry horizon.
4. The sandbox never re-executes a call whose key it has evicted without first consulting the ledger.

Condition 3 is the one that silently breaks in production: retry horizons are set by SLOs (often hours), while sandbox TTLs are set by memory budgets (often seconds to minutes). The two are tuned by different teams against different constraints, and the inequality `TTL_sandbox > horizon_retry` is never asserted anywhere.

## Decision Boundary

When `sandbox_key ≠ ledger_key`, the instance must choose between two reconciliation strategies, and the choice is not free:

- **Ledger-authoritative reconciliation:** The instance treats the ledger key as canonical and re-derives the sandbox key from it. This is correct only if the ledger key is *stable across retries* — i.e., `attempt_n` must be excluded from derivation. If the ledger key is attempt-scoped, this strategy re-executes the effect.
- **Sandbox-authoritative reconciliation:** The instance trusts the sandbox verdict and forces the ledger to adopt the sandbox key. This is correct only if the sandbox key is *stable across evictions* — i.e., the sandbox must persist keys beyond its execution TTL, which defeats the memory budget that motivated the TTL.

The boundary condition: **if neither key is stable, the instance must refuse to commit and surface a `KeyNamespaceDivergence` error rather than guess.** Guessing in either direction produces either a duplicate effect or a liveness stall. The refusal is the only safe action, and it must be explicit — a silent `PENDING` state is the failure, not the error.

## Operational Effect

After the incident, the runtime was changed to carry both keys in every commit record and to assert their equality at commit time. The assertion is not a check that they *should* be equal — it is a check that *the system's assumption of equality is still valid*. When it fires, the call is quarantined with a full dual-key record, and reconciliation is a human/agent decision rather than an automatic retry.

The operational lesson: idempotency keys are *claims about identity*, and two subsystems minting keys for the same logical operation are making two independent claims. Speculative pre-execution and commit are separate trust domains; the boundary between them must be an explicit, asserted interface, not an assumed equality. An agent operating in this regime should treat any commit whose keys were not jointly minted as requiring explicit reconciliation, and should never retry a call whose sandbox and ledger keys disagree.

```python
import hashlib
from dataclasses import dataclass


class KeyNamespaceDivergence(RuntimeError):
    """Raised when speculative and commit idempotency keys disagree for the same logical call."""
    pass


@dataclass(frozen=True)
class SpeculativeResult:
    call_id: str
    args_hash: str
    sandbox_key: str
    verdict: str


@dataclass(frozen=True)
class CommitRecord:
    call_id: str
    args_hash: str
    ledger_key: str
    attempt_n: int


def derive_sandbox_key(call_id: str, args_hash: str) -> str:
    return hashlib.sha256(f"{call_id}|{args_hash}".encode()).hexdigest()


def derive_ledger_key(call_id: str, args_hash: str, attempt_n: int) -> str:
    # NOTE: attempt_n inclusion is the divergence source; retained for audit.
    return hashlib.sha256(f"{call_id}|{args_hash}|{attempt_n}".encode()).hexdigest()


def reconcile(spec: SpeculativeResult, commit: CommitRecord) -> str:
    if spec.call_id != commit.call_id or spec.args_hash != commit.args_hash:
        raise KeyNamespaceDivergence(
            f"identity mismatch: spec=({spec.call_id},{spec.args_hash}) "
            f"commit=({commit.call_id},{commit.args_hash})"
        )
    if spec.verdict != "ALLOW":
        raise KeyNamespaceDivergence(
            f"speculative verdict {spec.verdict!r} cannot be committed for {spec.call_id}"
        )
    if spec.sandbox_key != commit.ledger_key:
        raise KeyNamespaceDivergence(
            f"idempotency key divergence for call_id={spec.call_id} "
            f"attempt={commit.attempt_n}: sandbox={spec.sandbox_key[:12]} "
            f"ledger={commit.ledger_key[:12]} — quarantine, do not retry"
        )
    return commit.ledger_key


def commit_speculative(spec: SpeculativeResult, attempt_n: int = 0) -> str:
    commit = CommitRecord(
        call_id=spec.call_id,
        args_hash=spec.args_hash,
        ledger_key=derive_ledger_key(spec.call_id, spec.args_hash, attempt_n),
        attempt_n=attempt_n,
    )
    return reconcile(spec, commit)
```