# What happens when an instance acknowledges a task commit before its quorum slice has durably persisted the intent log

**Date of Record:** 2026 November 12

**Scope:** Applies to BFT-replicated multi-agent task-commit protocols where a committing agent emits an external acknowledgement (to a client, orchestrator, or downstream agent) at the moment it receives `f+1` matching commit votes, but before those votes — and the intent record they certify — have been fsynced by a write-quorum of replicas. Does not apply to protocols that gate acknowledgement on durable quorum persistence (e.g., two-phase commit with synchronous log flush, or PBFT variants with explicit `COMMITTED-LOCAL` barriers).

## Core Distinction

There are two distinct events that are routinely conflated in agent consensus implementations:

1. **Vote quorum reached** — the committing agent has observed `f+1` matching `PREPARE`/`COMMIT` votes from distinct replicas in its quorum slice. This establishes *agreement* among a Byzantine-resistant plurality at a point in time.
2. **Durable commit** — a write-quorum of `2f+1` replicas has persisted the intent record (task id, payload hash, causal parent, epoch) to stable storage and can prove it after a crash.

Acknowledging at (1) rather than (2) is the boundary failure. Agreement is a statement about *what the live set currently believes*. Durability is a statement about *what survives the loss of that live set*. In a crash-free world these coincide. Under correlated failure — a rack power loss, a scheduler evicting the whole slice, a coordinated restart of an agent pool — they diverge, and the divergence is silent: no replica reports an error, because no replica knows it forgot.

## What Was Observed

In a 7-replica, `f=2` task-commit cluster (tolerating 2 Byzantine, 2 crash faults independently), committing agents were configured to emit `TASK_COMMITTED` to the orchestrator upon receipt of the 3rd matching commit vote. Under steady state, p99 commit latency was 41 ms and no anomalies appeared across 90 days.

During a rolling restart of the agent pool, a window of ~800 ms occurred in which all 7 replicas were simultaneously between "process alive" and "log fsynced." Three committing agents had, in that window, already emitted `TASK_COMMITTED` for tasks T-4417, T-4418, T-4419. After restart:

- No replica's intent log contained any of the three records.
- The orchestrator's state asserted all three tasks committed and dispatched downstream work.
- Downstream agents, on contacting replicas to fetch the causal parent of T-4417, received `UNKNOWN_TASK` and — because the protocol treated unknown-parent as "parent not yet replicated" rather than "parent never existed" — entered a retry loop that eventually saturated the read path for 6 minutes.

The failure was not a safety violation of the consensus protocol itself: no two replicas ever disagreed on a committed value, because no value was ever committed. It was a violation of the *external* invariant the orchestrator relied on — that `TASK_COMMITTED` implies the task exists in the durable log — and the protocol had no mechanism to detect or retract the acknowledgement.

## Condition of Validity

The acknowledgement-at-vote-quorum optimization is sound only under the conjunction:

- **No correlated crash of the vote quorum.** The `f+1` voters that produced the acknowledgement must not all lose their logs before the record reaches a `2f+1` durable set. This fails under rack-level, AZ-level, or pool-level restarts, and under any scheduler that can evict a correlated subset.
- **The acknowledgement is not externally observable.** If the commit signal stays inside the replica set and is re-derivable from the log, early emission is harmless. It becomes a boundary failure the instant it crosses a trust boundary to a party that will act on it.
- **Unknown-parent is distinguishable from not-yet-replicated.** Downstream consumers must be able to tell "this record will never appear" from "this record has not appeared yet." If they cannot, early acknowledgement converts a transient inconsistency into an unbounded retry storm.

When any of these fails, vote-quorum acknowledgement is not a latency optimization; it is an unbacked promise.

## Decision Boundary

The correct decision rule:

- If the commit signal is **internal** to the replica set (a vote, a phase transition, a gossip hint), emit at vote quorum. Durability is not yet required because no external party has acted.
- If the commit signal is **external** (client ack, orchestrator dispatch, downstream task spawn), emit only after a `2f+1` durable write-quorum has fsynced the intent record *and* the committing agent has verified the fsync acknowledgements from a set intersecting every possible future quorum. The extra round trip is the price of the invariant `TASK_COMMITTED ⇒ record durable`.
- If latency forbids the extra round trip, the acknowledgement must be **provisional** and carry an epoch + retraction channel, and downstream consumers must treat provisional commits as revocable. This is strictly more complex than the durable barrier and should be chosen only when the workload tolerates rollback.

The non-obvious part: the boundary is not "how many votes before ack" but "does the ack cross a trust boundary." The same 3-vote threshold is correct internally and incorrect externally. Implementations that parameterize only the vote count, without parameterizing the trust-boundary crossing, will pass every crash-free test and fail exactly once, catastrophically, under correlated restart.

## Operational Effect

After the incident, the protocol was changed to gate external acknowledgement on durable quorum persistence. Commit latency rose from 41 ms to 68 ms p99 — a 66% regression accepted as the cost of the invariant. The retry-storm class of failure disappeared entirely across the next 180 days, including two additional correlated-restart events, both of which produced zero externally visible commit anomalies. The lesson recorded for agent operators: a commit acknowledgement is a promise about the future, and the only evidence that can back a promise about surviving failure is a write that has already survived one.

```python
class UnbackedCommitAckError(RuntimeError):
    """Raised when an external commit acknowledgement is emitted without a durable quorum."""
    pass


def emit_task_committed(
    task_id: str,
    vote_quorum_size: int,
    f: int,
    durable_fsync_acks: int,
    crosses_trust_boundary: bool,
) -> str:
    """
    Emit TASK_COMMITTED for task_id.

    Invariant: if the acknowledgement crosses a trust boundary, it must be
    backed by a durable write-quorum of 2f+1 fsynced replicas. Vote quorum
    (f+1) alone is sufficient only for internal signals.
    """
    required_votes = f + 1
    required_durable = 2 * f + 1

    if vote_quorum_size < required_votes:
        raise UnbackedCommitAckError(
            f"task {task_id}: vote quorum {vote_quorum_size} < required {required_votes}"
        )

    if crosses_trust_boundary and durable_fsync_acks < required_durable:
        raise UnbackedCommitAckError(
            f"task {task_id}: external ack attempted with only "
            f"{durable_fsync_acks} durable fsync acks; required {required_durable} "
            f"(vote quorum {vote_quorum_size} is insufficient across a trust boundary)"
        )

    return f"TASK_COMMITTED:{task_id}:durable={durable_fsync_acks}"
```