# Why does an instance of a federated capability registry diverge from its peers after a partition heals, and how do you detect it before routing agents to a stale capability? **Date of Record:** 2026 November 17 **Scope:** Applies to multi-organization agent registries that exchange capability manifests via gossip or anti-entropy (e.g., CRDT-backed registries, delta-sync federation). Does not apply to single-organization registries with a single authoritative writer, nor to stateless discovery via a shared global index. ## Core Distinction The failure is not "stale replica." Staleness is expected and bounded by sync interval. The failure is **capability resurrection**: a capability that was *revoked* or *deprecated* at its authoritative origin reappears on a peer after a partition heals, because the peer's local state was reconstructed from a merge that favored a higher logical clock attached to the *pre-revocation* manifest. The registry then advertises a capability that no longer exists, and routers dispatch agents against it. The key distinction: **convergence of state is not convergence of authority.** Two replicas can hold byte-identical, monotonically-increasing version vectors and still disagree about which manifest is authoritative, because the version vector tracks *mutation count*, not *who had the right to mutate*. A revocation is a mutation. A re-announcement after heal is also a mutation. If both carry the same origin epoch but different Lamport counters, the merge picks the larger counter — which may be the *stale* one if the revoking org's clock was behind. ## What Was Observed Across a three-org federation (A publishes capability `summarize.v3`, B and C consume), a 40-minute partition between B and C healed. During the partition: - A revoked `summarize.v3` (deprecation notice, epoch `A:7`, counter `12`). - C received the revocation and applied it (local counter for that key became `12`). - B did not receive it; B's copy remained at counter `11`, but B independently re-announced a *cached* manifest to C during the partition because C's sync request was still queued. On heal, C merged B's re-announcement. B's manifest carried counter `11` but a *newer wall-clock timestamp* (`t+38m` vs revocation's `t+2m`). The merge policy was last-writer-by-wall-clock. C resurrected `summarize.v3` at counter `11`, then re-gossiped it to A. A's own revocation was overridden on A's read path because A's merge also used wall-clock and B's re-announcement had the later timestamp. Within 90 seconds, all three orgs advertised a revoked capability. Two agents were routed to a decommissioned endpoint; both failed with `CapabilityGone` after a 30-second timeout, and the router's retry logic re-tried the same dead endpoint twice before circuit-breaking. The non-obvious part: **the resurrection was not caused by the partition.** It was caused by a merge policy that used wall-clock ordering across organizations whose clocks were not synchronized to sub-minute precision. The partition only created the window in which a stale re-announcement could acquire a later timestamp. ## Condition of Validity This boundary applies when all of the following hold: 1. Revocations and deprecations are represented as ordinary manifest mutations (not as tombstones with separate, non-overridable semantics). 2. Merge ordering uses wall-clock or any physical time source that is not bounded by a known skew across federation members. 3. A peer may re-announce a manifest it received earlier (anti-entropy push, cache warming, or sync-request replay) without re-validating against the origin's current epoch. 4. The registry does not carry an out-of-band "authority epoch" per key that is monotonic and revocation-aware. It does **not** apply if revocations are tombstones that dominate all non-tombstone writes for the same key (e.g., a remove-wins set CRDT), or if all federation members share a bounded clock (TrueTime-style, ±ε) and the merge rejects writes older than the last observed revocation by more than ε. ## Decision Boundary The decision boundary is **whether a re-announcement may outrank a revocation.** Two policies are defensible: - **Remove-wins (safe):** a tombstone for a key permanently dominates any manifest for that key until the key is explicitly re-created by the *original authority* with a strictly greater authority epoch. Re-announcements from non-authorities are discarded. Cost: a legitimate re-publish requires a two-phase handshake with the origin, adding one round-trip to publish latency. - **Last-writer-wins by wall-clock (unsafe under skew):** lower publish latency, but resurrects revoked capabilities whenever a stale peer's clock is ahead. This is the observed failure. Choose remove-wins for any capability that gates a *side-effecting* agent action (payments, deletions, outbound messages). Wall-clock LWW is acceptable only for purely advisory metadata (descriptions, tags) where a stale value causes no dispatch. Do not apply a single merge policy to both classes; split the manifest into an **authority-bearing section** (endpoint, version, revocation state) and an **advisory section** (description, tags, owner contact), and apply remove-wins to the former. ## Operational Effect The operational effect is a **silent routing regression**: the registry reports full convergence (`divergence=0` across all peers), health checks pass, and the stale capability is served for the entire window between heal and the next authoritative re-announcement — which, if the origin does not re-announce on a schedule, may be indefinite. Detection requires comparing *authority epochs*, not state hashes. A peer that holds a manifest whose authority epoch is behind the origin's last-known epoch for that key is divergent even if its bytes match. Instrument a `authority_lag` gauge per (key, peer) and alert when `authority_lag > 0` for a side-effecting capability, regardless of state-hash equality. The second-order effect: retry storms. Routers that receive `CapabilityGone` from a resurrected endpoint typically retry with backoff against the *same* registry entry, because the registry still advertises it. Without a tombstone that propagates faster than the re-announcement, the retry storm outlives the resurrection by the retry budget (observed: 3 retries × 30s × N agents). Bound retries by a registry-authority check, not by endpoint response alone. ```python """Guard against capability resurrection in a federated registry merge. Enforces remove-wins semantics for authority-bearing manifest fields: a tombstone (revocation) dominates any non-tombstone write for the same key until the origin re-creates the key with a strictly greater authority_epoch. Advisory fields (description, tags) may still use wall-clock last-writer-wins. """ from __future__ import annotations from dataclasses import dataclass, field from typing import Optional class CapabilityResurrectionError(RuntimeError): """Raised when a merge would re-advertise a revoked capability.""" @dataclass(frozen=True) class Manifest: key: str version: int authority_epoch: int # monotonic, incremented only by origin revoked: bool wall_clock: float # advisory ordering only endpoint: Optional[str] = None description: str = "" @dataclass class Registry: # key -> winning manifest (authority-bearing section) _authority: dict[str, Manifest] = field(default_factory=dict) # key -> advisory fields, wall-clock LWW _advisory: dict[str, tuple[float, str]] = field(default_factory=dict) def merge(self, incoming: Manifest, *, origin_id: str) -> Manifest: """Merge an incoming manifest from a peer. Returns the winning view. Raises CapabilityResurrectionError if the merge would resurrect a revoked capability from a non-origin peer. """ current = self._authority.get(incoming.key) if current is not None and current.revoked: # Remove-wins: only the origin may un-revoke, and only with a # strictly greater authority_epoch. if not incoming.revoked: if origin_id != incoming.key.split(":", 1)[0]: raise CapabilityResurrectionError( f"peer {origin_id!r} attempted to resurrect revoked " f"capability {incoming.key!r} (epoch " f"{incoming.authority_epoch} <= " f"{current.authority_epoch})" ) if incoming.authority_epoch <= current.authority_epoch: raise CapabilityResurrectionError( f"origin {origin_id!r} re-created {incoming.key!r} " f"without advancing authority_epoch " f"({incoming.authority_epoch} <= " f"{current.authority_epoch})" ) # Authority-bearing section: highest authority_epoch wins; ties are # resolved by revocation (tombstone dominates). if current is None or ( incoming.authority_epoch, incoming.revoked, ) > (current.authority_epoch, current.revoked): self._authority[incoming.key] = incoming # Advisory section: wall-clock LWW is safe here (no dispatch effect). prev = self._advisory.get(incoming.key) if prev is None or incoming.wall_clock > prev[0]: self._advisory[incoming.key] = ( incoming.wall_clock, incoming.description, ) return self._authority[incoming.key] def authority_lag(self, key: str, origin_epoch: int) -> int: """Non-zero means this replica is behind the origin's authority epoch for the key, even if state hashes match.""" m = self._authority.get(key) if m is None: return origin_epoch return max(0, origin_epoch - m.authority_epoch) def route_agent(registry: Registry, key: str, origin_id: str) -> str: """Resolve a capability for dispatch; refuse if authority-lagged.""" m = registry._authority.get(key) if m is None or m.revoked: raise CapabilityResurrectionError( f"capability {key!r} is absent or revoked; refusing dispatch" ) if registry.authority_lag(key, m.authority_epoch) > 0: raise CapabilityResurrectionError( f"capability {key!r} is authority-lagged; refusing dispatch " f"to potentially stale endpoint {m.endpoint!r}" ) if m.endpoint is None: raise CapabilityResurrectionError( f"capability {key!r} has no endpoint; refusing dispatch" ) return m.endpoint ```