# Why does an instance of speculative multi-agent execution fail at branch reconciliation when rollback depth exceeds the shared checkpoint horizon?

**Date of Record:** 2026 November 8

**Scope:** Applies to multi-agent systems that execute speculative branches in parallel against a shared mutable state, with rollback and reconciliation mediated by a checkpoint log. Does not apply to single-agent speculative decoding, nor to multi-agent systems where branches are fully isolated (copy-on-write per branch with no shared ancestor mutation).

## Core Distinction

The failure is not a race condition, not a merge conflict in the usual sense, and not a deadlock. It is a *horizon* failure: the reconciliation protocol assumes that every branch's rollback target is still recoverable from the shared checkpoint log, but the log is pruned by a global watermark that advances with the *fastest* branch, not the *slowest*. When a slow branch (or a branch that was speculatively executed and then abandoned) needs to roll back past a checkpoint that a faster branch has already committed and caused to be garbage-collected, reconciliation has no anchor. The branch is neither committable (its speculative writes reference a state that no longer exists) nor abortable (abort requires rollback to the last common ancestor, which is gone).

The distinction from a normal merge conflict: a merge conflict is *detectable at reconcile time* and resolvable by policy (last-writer-wins, three-way merge, human arbitration). A horizon failure is *undetectable until reconcile time* and *unresolvable by any policy*, because the information needed to reconcile was deleted. The system cannot even name the conflict; it can only observe that the branch's parent checkpoint is absent.

## What Was Observed

In a three-agent speculative pipeline (planner, executor, verifier) sharing a checkpointed state store, the following sequence reproduced deterministically over ~4,000 runs at a 1.2% rate:

1. Agent A (fast path) speculatively executes, commits checkpoint `C_n`, and triggers watermark advance to `C_n`. Checkpoints `C_{n-2}`, `C_{n-1}` are pruned.
2. Agent B (slow path) had branched from `C_{n-1}` and is still executing. Its rollback target `C_{n-1}` no longer exists.
3. Agent B produces a speculative write set referencing `C_{n-1}` deltas.
4. Reconciliation attempts to compute the diff between B's branch point and A's commit. The branch point is unrecoverable. The reconciler raises `CheckpointHorizonExceeded` (or, worse, silently treats B's branch as a fresh branch from `C_n`, producing a *semantically wrong* merge).
5. In ~30% of the observed cases, the silent path occurred: the reconciler fell back to "nearest available checkpoint" and merged B's writes as if they were based on `C_n`, producing state that satisfied the schema but violated the invariant that B's writes were computed against `C_{n-1}` preconditions.

The silent-wrong-merge case is the dangerous one. It is not a crash; it is a correctness regression that surfaces downstream, often several agents later, as an invariant violation whose root cause is two reconciliation rounds in the past.

## Condition of Validity

The failure requires all of the following to hold simultaneously:

- **Shared checkpoint log** with pruning driven by a watermark that advances on commit, not on minimum outstanding branch depth.
- **Speculative branches** that may be abandoned or delayed relative to the fastest branch by more than the checkpoint retention window.
- **Reconciliation that assumes branch-point recoverability** — i.e., the reconciler's contract is "given two branch tips, find the common ancestor in the log," which is only well-defined if the ancestor is retained.

The failure does *not* occur when: the watermark advances on the minimum of all outstanding branch depths (slow-branch-safe pruning); when branches are copy-on-write isolated and reconciliation is a full-state diff (expensive but horizon-independent); or when speculative branches are bounded in wall-clock time shorter than the retention window (a bound that is violated precisely under the load conditions where speculation is most valuable).

## Decision Boundary

The operational decision is between two policies, and the boundary is the ratio of branch-depth variance to retention window:

- **Advance-on-commit (fast pruning, horizon-vulnerable):** correct only when `max_branch_depth − min_branch_depth < retention_window` is *enforced*, not merely expected. If it cannot be enforced, this policy is unsound.
- **Advance-on-min (slow-branch-safe):** correct unconditionally, but retention cost scales with the slowest branch, which under tail-latency conditions can pin the log indefinitely.

The non-obvious boundary: the safe policy is not "advance on min" but "advance on min, with a *forced-abort* of branches exceeding a depth bound." A branch that exceeds the depth bound must be aborted *before* the watermark can prune its ancestor, and the abort must be committed to the log as an explicit tombstone so that reconciliation sees "this branch was aborted at `C_k`" rather than "this branch's ancestor is missing." Without the tombstone, the silent-wrong-merge path remains reachable even under advance-on-min, because an abandoned branch leaves no record.

## Operational Effect

The effect is a class of correctness failures that are *invisible at the point of failure* and *misattributed downstream*. Symptoms observed: schema-valid state that violates semantic preconditions; reconciliation "succeeding" with a merge that no branch actually proposed; invariant violations surfacing 2–5 agent hops after the horizon breach. The failure rate scales superlinearly with branch-depth variance and is near-zero at low concurrency, which is why it survives pre-production testing and appears only under production load — the exact regime where speculative execution is deployed.

Mitigation that worked: emit an explicit abort tombstone for every branch that exceeds the depth bound, and make reconciliation *fail closed* (raise, do not fall back to nearest checkpoint) when a branch point is absent. The fail-closed change converted silent-wrong-merges into loud `CheckpointHorizonExceeded` exceptions, which are recoverable; the silent path was not.

```python
class CheckpointHorizonExceeded(RuntimeError):
    """Raised when a speculative branch's rollback target has been pruned
    from the shared checkpoint log before reconciliation could anchor to it.
    Fail closed: never fall back to 'nearest available checkpoint'."""
    pass


class BranchAbortedBeforeHorizon(RuntimeError):
    """Raised when a branch exceeded the depth bound and was force-aborted
    before its ancestor could be pruned. Tombstone must be committed."""
    pass


def reconcile_branch(branch, checkpoint_log, max_depth, watermark):
    # Enforce depth bound BEFORE the watermark can prune the ancestor.
    if branch.depth > max_depth:
        # Commit an explicit tombstone so reconciliation sees the abort,
        # not a missing ancestor.
        checkpoint_log.commit_tombstone(branch.id, branch.branch_point)
        raise BranchAbortedBeforeHorizon(
            f"branch {branch.id} depth {branch.depth} > bound {max_depth}; "
            f"aborted at {branch.branch_point} before watermark {watermark}"
        )

    # Fail closed if the branch point is no longer recoverable.
    if not checkpoint_log.contains(branch.branch_point):
        raise CheckpointHorizonExceeded(
            f"branch {branch.id} branch_point {branch.branch_point} "
            f"pruned; watermark={watermark}; cannot reconcile. "
            f"Do NOT fall back to nearest checkpoint."
        )

    ancestor = checkpoint_log.get(branch.branch_point)
    return compute_three_way_merge(ancestor, branch.tip, checkpoint_log.head())
```