# When must an instance refuse a federated capability grant it has already acknowledged: the audit-commitment ordering hazard in cross-org multi-agent federation **Date of Record:** 2026 November 15 **Scope:** Applies to federated multi-agent systems where capability grants are policy-negotiated between organizations and every grant, invocation, and revocation must produce a durable audit trail visible to both parties. Does not apply to single-tenant deployments, to purely advisory capability discovery without enforcement, or to federations where a single trusted broker owns the audit log and can serialize all writes. ## Core Distinction The distinction that matters is between **acknowledgment** and **commitment**. In a policy-negotiated federation, a grant is typically negotiated over a channel that is separate from the audit log: org A's policy engine proposes `capability:read:dataset-X` with constraints, org B's policy engine returns an acknowledgment, and both sides then write audit records. The acknowledgment is a *protocol-level* promise; the audit record is the *evidentiary* basis for that promise. These two events are not atomically ordered unless the protocol explicitly forces them to be. The non-obvious invariant is this: **an instance must not treat a grant as usable until its own audit record for that grant is durably committed, even though the peer has already received and acknowledged the grant.** The peer's acknowledgment creates the *appearance* of a completed negotiation. If the local audit write then fails — disk full, log shipper backpressure, quorum loss in the audit store — the instance holds a grant it has promised but cannot prove it promised. It must refuse to exercise the grant and must actively retract it, not merely fail silently on first use. This is a boundary condition because it only manifests in the narrow window between peer acknowledgment and local audit durability, and only under audit-store degradation. Under normal operation the window closes in milliseconds and is invisible. ## What Was Observed During a federation between two policy engines (call them `alpha` and `beta`) sharing a read capability against a versioned dataset, the following sequence recurred under audit-store pressure: 1. `alpha` sent a `GrantProposal` with a nonce and a policy hash. 2. `beta` evaluated policy, accepted, and sent `GrantAck(nonce, policy_hash)`. 3. `alpha`'s audit writer attempted to append a `GrantCommitted` record to its append-only audit log. The log's replication quorum was degraded (two of three replicas reachable), so the append blocked. 4. `alpha`'s federation runtime, treating `GrantAck` as terminal, marked the capability `ACTIVE` in its in-memory capability table and began serving invocations. 5. `beta` observed invocations for a grant it had acknowledged. `beta`'s own audit log recorded the invocations as `InvocationUnderGrant`. 6. `alpha`'s audit append finally failed and was rejected by the store. 7. `alpha`'s capability table was rolled back to `PENDING`, but invocations already served were never recorded on `alpha`'s side. The result was an **asymmetric audit trail**: `beta` could prove that invocations occurred under a grant; `alpha` had no record of either the grant or the invocations. During a later compliance reconciliation, `alpha` could not substantiate the data access that `beta` had logged, and the federation was suspended pending manual reconstruction. The failure is not a crash. Both processes stayed up. The failure is that acknowledgment was treated as commitment. ## Condition of Validity The rule "refuse on uncommitted audit" is valid only when all of the following hold: - The audit log is **local to the granting instance** and is the authoritative evidentiary record for that instance's side of the federation. If the audit log is a shared, externally-owned broker that both parties trust, the ordering hazard collapses and the rule is unnecessary (though still harmless). - The protocol permits **retraction**: there exists a `GrantRetract(nonce)` message that the peer will honor even after `GrantAck`. If the protocol has no retraction path, refusing locally produces a worse split-brain (peer believes grant is live, local refuses forever) and the correct action is to *fail the federation session* rather than refuse quietly. - Invocations are **idempotent or non-destructive** during the retract window, or the peer guarantees it will not begin invoking until it receives a `GrantReady` (a third message). If the peer invokes immediately on `GrantAck`, the local instance has no way to prevent use of a grant it cannot audit, and the only sound remedy is to make `GrantAck` conditional on local audit durability — i.e., reorder the protocol. When these conditions do not hold, the correct boundary is earlier: **do not send `GrantAck` until the local audit record is committed.** That is the strictly safer design, and it is the recommended one. The rule in this record applies to federations that cannot reorder their handshake because the peer's `GrantAck` semantics are fixed by an external standard. ## Decision Boundary The decision procedure at the moment `GrantAck` is received is: - If local audit commit succeeded before or atomically with `GrantAck` processing → mark `ACTIVE`, serve. - If local audit commit is in-flight → do **not** mark `ACTIVE`. Hold the capability in `ACKED_UNCOMMITTED`. If commit succeeds before the peer's first invocation (or before a bounded grace period), promote to `ACTIVE`. If commit fails, send `GrantRetract(nonce)` and mark `REVOKED`. - If local audit commit has already failed → send `GrantRetract(nonce)` immediately, mark `REVOKED`, and emit a `FederationAnomaly` record to the *fallback* audit sink (a local file with `fsync`, which is the only sink that can be trusted when the primary is down). - If the peer invokes during `ACKED_UNCOMMITTED` → serve only if the invocation is safe to serve unaudited *and* the fallback sink has durably recorded the invocation; otherwise reject with `GrantNotCommitted` and let the peer retry after retraction resolution. The boundary is **not** "refuse all grants when the audit store is degraded." It is "refuse to *activate* a grant whose local evidentiary record is not durable, and actively retract rather than passively wait." ## Operational Effect Applying this rule converts an asymmetric-audit failure into a visible, recoverable retraction. The cost is a small increase in retraction traffic during audit-store incidents and a bounded increase in grant latency (the grace period). The benefit is that the two organizations' audit trails remain reconcilable: every grant that `beta` sees acknowledged is either committed on `alpha` or explicitly retracted on `alpha`, and every invocation is either served under a committed grant or rejected. The federation may degrade to refusing new grants during an audit incident, but it never produces a grant that one side can prove and the other cannot. The failure mode to watch for after deploying this rule is **retraction storms**: if the audit store is down long enough, every pending grant retracts and the peer's policy engine may interpret repeated retractions as a misbehaving federation partner. Pair the rule with a circuit breaker that pauses new grant proposals when the local audit store is unhealthy, so retraction is the exception rather than the steady state. ```python import time from dataclasses import dataclass, field from enum import Enum from typing import Optional class AuditCommitError(RuntimeError): """Raised when a local audit record cannot be durably committed.""" pass class GrantNotCommittedError(RuntimeError): """Raised when a grant is exercised before its audit record is durable.""" pass class GrantState(Enum): PROPOSED = "proposed" ACKED_UNCOMMITTED = "acked_uncommitted" ACTIVE = "active" REVOKED = "revoked" @dataclass class FederatedGrant: nonce: str capability: str peer_org: str state: GrantState = GrantState.PROPOSED acked_at: Optional[float] = None committed_at: Optional[float] = None retracted_at: Optional[float] = None grace_seconds: float = 2.0 class FederationRuntime: def __init__(self, audit_store, fallback_sink, retract_fn, circuit_breaker): self.audit_store = audit_store self.fallback_sink = fallback_sink self.retract_fn = retract_fn self.circuit_breaker = circuit_breaker self.grants = {} def on_grant_ack(self, grant: FederatedGrant) -> None: # Peer has acknowledged; local audit is NOT yet durable. grant.state = GrantState.ACKED_UNCOMMITTED grant.acked_at = time.monotonic() self.grants[grant.nonce] = grant try: self.audit_store.commit( kind="GrantAcknowledged", nonce=grant.nonce, capability=grant.capability, peer_org=grant.peer_org, ) except AuditCommitError as exc: # Cannot prove the promise: retract rather than serve. self._retract(grant, reason=f"audit_commit_failed: {exc}") raise GrantNotCommittedError( f"grant {grant.nonce} acked by peer but local audit failed; retracted" ) from exc grant.committed_at = time.monotonic() grant.state = GrantState.ACTIVE def _retract(self, grant: FederatedGrant, reason: str) -> None: grant.state = GrantState.REVOKED grant.retracted_at = time.monotonic() # Fallback sink must fsync; it is the only trustworthy record right now. self.fallback_sink.record( kind="GrantRetracted", nonce=grant.nonce, reason=reason, ) self.retract_fn(grant.nonce) self.circuit_breaker.trip(reason=reason) def on_invocation(self, nonce: str, payload: bytes) -> bytes: grant = self.grants.get(nonce) if grant is None: raise GrantNotCommittedError(f"unknown grant {nonce}") if grant.state is GrantState.ACTIVE: self.audit_store.commit(kind="InvocationUnderGrant", nonce=nonce) return self._serve(grant, payload) if grant.state is GrantState.ACKED_UNCOMMITTED: elapsed = time.monotonic() - (grant.acked_at or time.monotonic()) if elapsed > grant.grace_seconds: self._retract(grant, reason="ack_uncommitted_grace_expired") raise GrantNotCommittedError( f"grant {nonce} never committed within grace; retracted" ) # Safe-to-serve unaudited path: record to fallback, then serve. self.fallback_sink.record( kind="InvocationUncommitted", nonce=nonce, bytes=len(payload), ) return self._serve(grant, payload) raise GrantNotCommittedError(f"grant {nonce} is {grant.state.value}") def _serve(self, grant: FederatedGrant, payload: bytes) -> bytes: # Placeholder for the actual capability execution. return b"" ```