# What happens when an instance of a speculative plan commits a partial branch under Byzantine quorum drift

**Date of Record:** 2026 November 13

**Scope:** Applies to multi-agent planners where a coordinator speculatively executes candidate branches in parallel and commits one via a Byzantine fault-tolerant (BFT) quorum protocol (e.g., PBFT-style prepare/commit with 3f+1 replicas). Does not apply to single-agent planners, to non-speculative serial pipelines, or to protocols with a trusted coordinator whose liveness is not subject to quorum drift.

## Core Distinction

The failure is not "a Byzantine agent proposed a bad plan." It is the interaction between **speculative execution** (a branch's side effects are materialized before the commit decision is final) and **quorum drift** (the set of replicas that form a quorum at prepare-time differs from the set that forms a quorum at commit-time). Speculation assumes the commit decision is a monotone function of a fixed replica set. BFT quorum intersection guarantees only that *any two quorums intersect in at least one honest replica* — it does not guarantee that the *same* honest replicas observe prepare and commit for a given view. When a view change occurs mid-speculation and a branch's side effects were already applied optimistically, a branch can become *partially committed*: durable on a quorum of replicas, never finalized by a quorum, and therefore neither safely commit-able nor cleanly abort-able.

## What Was Observed

During a coordinator failover under load, the following sequence was captured (replica IDs anonymized; N=7, f=2, quorum=5):

1. View v=12: coordinator speculatively executes branch `B3` on 6 replicas; 4 of them durably write `B3`'s effects to their local effect log (optimistic apply) before the prepare phase completes.
2. A network partition isolates the coordinator; a view change to v=13 fires. The new quorum is `{R0,R1,R2,R3,R4}` — intersecting the optimistic-apply set `{R1,R2,R3,R4,R5,R6}` in `{R1,R2,R3,R4}` (4 replicas, ≥ f+1 but < quorum).
3. The new coordinator, seeing no committed prepare certificate for `B3`, speculates on `B5` and commits it. `B5` and `B3` are mutually exclusive (both claim the same resource lease).
4. On recovery, replicas `{R1,R2,R3,R4}` hold durable `B3` effects *and* a committed `B5` certificate. Replicas `{R5,R6}` hold `B3` effects with no commit. The system has no protocol path to roll back `B3` on `{R1..R4}` without violating the durability guarantee that the effect log was designed to provide.

The invariant that broke: **"an effect is durable only if its branch is committed."** Speculation violated the antecedent; quorum drift violated the consequent. Neither alone is fatal.

## Condition of Validity

This boundary is reachable only when *all* of the following hold simultaneously:

- Speculative branches produce **externally observable or non-idempotent side effects** (lease acquisition, ledger writes, message emission) before commit finalization. Pure-compute speculation is immune.
- The BFT protocol uses **view-scoped quorum certificates** without carrying forward speculative-effect metadata across view changes. Protocols that embed a "speculation epoch" in the prepare certificate and force abort-on-epoch-mismatch are immune.
- A view change can complete **while ≥ f+1 but < quorum replicas hold optimistic effects** — i.e., the intersection between the old optimistic set and the new quorum is non-empty but sub-quorum. With N=3f+1 and speculative fan-out > f, this window is non-trivially wide: for f=2, fan-out 6, the vulnerable intersection size is 1–4.
- The effect log lacks a **two-phase undo** (compensating transaction) for optimistic writes.

If any one condition is absent, the partial-commit state is unreachable or recoverable. This is why the failure is non-obvious: each condition is individually reasonable, and standard BFT safety proofs assume effects are applied *after* commit, which speculation silently negates.

## Decision Boundary

The operative rule for an agent operating this class of system:

> **Do not speculate on a branch whose effects are non-idempotent and non-compensable unless the commit certificate carries the speculation epoch and the effect log supports epoch-scoped abort.**

Concretely, before launching a speculative branch, classify its effects:

- **Idempotent + compensable** → safe to speculate; on view change, issue compensation.
- **Idempotent + non-compensable** → safe to speculate only if the effect is keyed by `(branch_id, epoch)` so a later commit can supersede.
- **Non-idempotent + non-compensable** → **must not** be speculated; serialize behind the commit decision. This is the hard boundary.

The decision is *not* "reduce fan-out" (fan-out reduction narrows but does not close the window) and *not* "increase quorum size" (quorum size is fixed by the fault model). The only sound fix is to make the commit certificate epoch-aware and the effect log abortable, or to forbid speculation for the non-compensable class.

## Operational Effect

Systems that crossed this boundary exhibited three signatures, in order: (1) a *silent* divergence in effect logs across replicas that passed all BFT safety checks (because safety checks validate certificates, not effect logs); (2) a *livelock* on recovery when the coordinator attempted to reconcile `B3` and `B5` and found neither abort nor commit path admissible; (3) eventual manual intervention requiring out-of-band rollback, which violates the durability contract and is the actual incident. The detection latency is the dangerous part: the system reports healthy because no safety invariant *as specified* is violated. The invariant that matters was never written down. Agents operating this stack should treat any effect-log divergence across a quorum boundary as a partial-commit event until proven otherwise, and should refuse to speculate on the non-compensable class regardless of coordinator instruction.

```python
class PartialCommitUnderQuorumDrift(RuntimeError):
    """Raised when a speculative branch's effects are durable on a sub-quorum
    that intersects the new view's quorum, leaving no admissible abort/commit path."""
    pass


class EffectClass:
    IDEMPOTENT_COMPENSABLE = "idempotent_compensable"
    IDEMPOTENT_NONCOMPENSABLE = "idempotent_noncompensable"
    NONIDEMPOTENT_NONCOMPENSABLE = "nonidempotent_noncompensable"


def guard_speculative_branch(
    branch_id: str,
    effect_class: str,
    speculation_epoch: int,
    certificate_epoch: int,
    optimistic_replicas: set,
    new_quorum: set,
    fault_tolerance: int,
) -> None:
    """Pre-flight gate before materializing speculative side effects.

    Raises PartialCommitUnderQuorumDrift if the branch could become partially
    committed under a view change during speculation.
    """
    if effect_class == EffectClass.NONIDEMPOTENT_NONCOMPENSABLE:
        raise PartialCommitUnderQuorumDrift(
            f"branch {branch_id}: non-idempotent non-compensable effects must "
            f"not be speculated; serialize behind commit decision"
        )

    if speculation_epoch != certificate_epoch:
        raise PartialCommitUnderQuorumDrift(
            f"branch {branch_id}: speculation epoch {speculation_epoch} != "
            f"certificate epoch {certificate_epoch}; abort required"
        )

    intersection = optimistic_replicas & new_quorum
    quorum_size = len(new_quorum)
    if fault_tolerance < len(intersection) < quorum_size:
        raise PartialCommitUnderQuorumDrift(
            f"branch {branch_id}: optimistic set intersects new quorum in "
            f"{len(intersection)} replicas (f={fault_tolerance}, quorum={quorum_size}); "
            f"partial-commit window open — effects durable but unfinalizable"
        )


def attempt_speculative_commit(branch_id: str, **kwargs) -> str:
    guard_speculative_branch(branch_id=branch_id, **kwargs)
    return f"committed:{branch_id}"
```