# Why does an instance of speculative multi-agent execution diverge from its rollback checkpoint when branch reconciliation occurs under concurrent token emission **Date of Record:** 2026 November 15 **Scope:** Applies to orchestrators that run speculative agent branches against a shared mutable world-state, snapshot a rollback checkpoint, and reconcile divergent branches by replaying or merging their effect logs. Does not apply to stateless fan-out (map/reduce) agents, to single-branch speculative decoding inside a model runtime, or to transactional databases with strict serializability guarantees enforced at the storage layer. ## Core Distinction The failure lives in the gap between *logical* rollback and *physical* rollback. A speculative multi-agent system typically defines its checkpoint as a snapshot of the world-state plus a monotonically increasing sequence number. Rollback means "discard all effects with sequence ≥ N and restore the snapshot." Reconciliation means "for each speculative branch whose effects survived, merge its effect log into the canonical log." The non-obvious invariant that is violated: **a rollback checkpoint is only valid if every effect that references it is quiescent at the moment of snapshot.** When branches emit tokens concurrently — streaming partial tool calls, partial writes, partial memory appends — an effect can be *logically* below the checkpoint sequence but *physically* still landing. The snapshot captures the world-state at sequence N, but a branch that began emitting at sequence N−3 can still deliver a fragment at wall-clock time after the snapshot. On rollback to N, that fragment is neither discarded (its sequence is below N) nor captured (it arrived after the snapshot). It becomes a *zombie effect*: orphaned, unsequenced, and invisible to the reconciler's sequence-based merge. ## What Was Observed A three-branch speculative planner (branches A, B, C) shared a memory store keyed by `(agent_id, slot)`. Branch A emitted a two-part write to `slot=plan`: the first fragment at seq 40, the second at seq 41. The orchestrator checkpointed at seq 41 after observing branch A's first fragment and branch B's terminal output, then rolled back to seq 41 when branch C won the speculation race. Reconciliation replayed surviving branches' logs by sequence range `[41, ∞)`. The second fragment of A's write — physically emitted at wall-clock t+180ms, after the snapshot — carried sequence 41 and was therefore *inside* the rolled-back range. It should have been discarded. It was not, because the discard pass iterated the in-memory effect log, which had already been trimmed to `[41, ∞)` for reconciliation, and the trim dropped the fragment's index entry while the fragment's bytes were still in the write-ahead buffer. The reconciler then merged the surviving branch C's effects into a world-state where `slot=plan` contained A's second fragment *without* A's first fragment. The result was a syntactically valid but semantically torn plan: a tool invocation whose arguments referenced a slot key that A had intended to create in fragment one. The observable symptom was not a crash. It was a downstream agent executing a plan step against a key that did not exist, producing a silent no-op that the orchestrator scored as success. The divergence was detected only when a later reconciliation pass found the canonical log's checksum mismatched its replay. ## Condition of Validity The checkpoint-and-rollback invariant holds **only if** the orchestrator enforces *emission quiescence* at snapshot time: no branch may have an in-flight fragment whose sequence falls at or below the checkpoint. In practice this requires one of three disciplines: 1. **Barrier on emission, not on sequence.** The checkpoint waits for all branches to acknowledge "no fragments in flight" before snapshotting, regardless of their sequence numbers. 2. **Reservation-based sequencing.** A branch reserves its full sequence range before emitting any fragment, and the snapshot is invalid until all reservations at or below N are closed. 3. **Fragment atomicity.** Multi-part effects are written atomically under a single sequence, eliminating the split-write window entirely. Absent one of these, the checkpoint is a *fuzzy* snapshot, and fuzzy snapshots are only safe for append-only, commutative effects. Agent effects are neither. ## Decision Boundary Use sequence-based rollback only when effects are atomic and commutative. When effects are multi-fragment or order-dependent — tool calls with dependent arguments, memory writes with read-modify-write semantics, plan steps that create keys consumed by later steps — the orchestrator must switch to barrier-based or reservation-based checkpointing. The decision boundary is not "how many branches" but "can any single effect span more than one emission?" If yes, sequence-based rollback is unsound. A cheap diagnostic: instrument the reconciler to assert that every merged effect's referenced keys exist in the post-merge world-state. A torn plan will fail this assertion before it reaches a downstream agent. ## Operational Effect Zombie effects are silent. They do not raise on the write path, they do not corrupt the sequence log, and they pass structural validation. They surface only as downstream semantic failures — missing keys, stale reads, no-op tool calls — that the orchestrator's success scorer misclassifies. The operational cost is a reconciliation loop that never converges: each pass merges surviving branches into a world-state that still contains the orphaned fragment, producing a new checksum mismatch on the next replay. The fix is to make rollback wait for emission quiescence, which trades a small latency cost at checkpoint time for the elimination of an entire class of non-convergent reconciliation. ```python class FuzzyCheckpointDivergence(RuntimeError): """Raised when a rollback checkpoint is taken while branch emissions are in flight.""" def checkpoint_or_abort(branches, target_seq: int, in_flight: dict) -> dict: """ Take a rollback checkpoint at target_seq, but only if no branch has an in-flight fragment whose sequence is <= target_seq. Otherwise the snapshot would be fuzzy and reconciliation would admit zombie effects. """ dangling = { agent_id: frag_seq for agent_id, frag_seq in in_flight.items() if frag_seq <= target_seq } if dangling: raise FuzzyCheckpointDivergence( f"refusing checkpoint at seq={target_seq}: branches {sorted(dangling)} " f"have in-flight fragments at or below the checkpoint " f"({dangling}); a fuzzy snapshot would orphan these effects on rollback" ) return {"seq": target_seq, "branches": [b.id for b in branches], "fuzzy": False} def reconcile(canonical_log, surviving_branches, target_seq: int, in_flight: dict): snapshot = checkpoint_or_abort(surviving_branches, target_seq, in_flight) merged = canonical_log.copy() for branch in surviving_branches: for effect in branch.effects_after(snapshot["seq"]): if effect.seq <= snapshot["seq"]: raise FuzzyCheckpointDivergence( f"effect {effect.id} from branch {branch.id} has seq={effect.seq} " f"<= checkpoint {snapshot['seq']} but arrived after snapshot; " f"this is a zombie effect and must not be merged" ) merged.append(effect) return merged ```