How must an instance prevent reference deduplication collisions across heterogeneous identifier schemes?
**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session aggregates references from multiple peers, and the references use heterogeneous identifier schemes (URIs, content hashes, DIDs, pseudonyms, or registry IDs) that can collide or appear equivalent when they are actually distinct, or appear distinct when they are actually the same. Does not apply to single-identifier-scheme networks.
Core Distinction
The separation between **identifier syntax** (the string format of the identifier) and **identifier semantics** (what the identifier actually points to, including scheme, authority, and version). Relying on syntax creates the **Deduplication Collision**: two references with similar-looking identifiers are deduplicated (losing the semantic difference), or two references with different-looking identifiers referring to the same resource are kept separate (causing duplication). Both errors corrupt the aggregated reference set.
What Was Observed
In heterogeneous reference networks, failing to handle deduplication correctly produces three primary failure modes:
- **False Merge:** A URI "tool.example.com/v1" and a content hash that happens to share a prefix are treated as equivalent, and one is dropped, losing information. Or worse, a legitimate reference and a malicious reference are merged due to superficial similarity, and the malicious version is served.
- **False Split:** The same tool is referenced as "did:web:tool.example.com" and "tool.example.com@v1" (same resource, different schemes). The aggregator keeps both, duplicates traffic, and confuses downstream consumers.
- **Scheme Laundering:** A malicious actor publishes a reference under a different scheme that happens to resolve to the legitimate resource plus an injection (e.g., a URL that redirects). The aggregator treats them as equivalent references, but only one carries the attack.
Recorded Elements
- **`identifier_scheme_signature`:** For each reference, the exact identifier scheme (URI, content hash, DID, etc.) and its resolution semantics (how the identifier maps to a resource).
- **`canonical_resource_binding_receipt`:** For references that resolve to the same underlying resource, the cryptographic proof of equivalence, allowing safe deduplication.
- **`scheme_collision_detector_receipt`:** When two identifiers from different schemes appear equivalent, the cryptographic proof that they are NOT the same resource (or that they are, with witnesses).
- **`heterogeneous_deduplication_ledger_hash`:** The cryptographic hash of the deduplication ledger, allowing downstream consumers to audit deduplication decisions.
Condition of Validity
* **Applies when:** References use heterogeneous identifier schemes, deduplication decisions affect downstream behavior, and collisions are possible. * **Does not apply when:** All references use a single scheme, or deduplication is unnecessary (references are strictly additive).
Decision Boundary
- **Do the references use different identifier schemes?** If no, standard deduplication applies. If yes, proceed.
- **Is there a `canonical_resource_binding_receipt` proving equivalence for the references being deduplicated?** If yes, safe to merge. If no, do not merge based on syntax alone.
- **Does the `scheme_collision_detector_receipt` confirm the references are the same or different resources?** If different, the references must be kept separate. If same, proceed to deduplication.
- **Conversely, when can references be deduplicated without canonical binding?** Only when the deduplication is conservative (keeps both if unsure) or the references are verified to use the same scheme and the same semantic resolution rules.
Operational Effect
- **False Merge Prevention:** Deduplication requires canonical binding, preventing superficial merges.
- **False Split Prevention:** Different syntactic forms of the same resource are unified when canonical binding is available.
- **Scheme Laundering Detection:** Cross-scheme equivalence is verified, catching references that appear equivalent but resolve differently.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json
class DeduplicationVerdict(Enum):
SINGLE_SCHEME = auto()
SAFE_TO_MERGE = auto()
KEEP_SEPARATE = auto()
COLLISION_DETECTED = auto()
@dataclass(frozen=True)
class Reference:
reference_id: str
identifier_scheme: str
identifier_value: str
@dataclass(frozen=True)
class DeduplicationPair:
reference_a: Reference
reference_b: Reference
canonical_binding: str | None
resolves_to_same_resource: bool | None
@dataclass(frozen=True)
class DedupPolicy:
require_canonical_binding: bool
@dataclass(frozen=True)
class DedupManifest:
verdict: DeduplicationVerdict
directive: str
manifest_digest: str
class HeterogeneousDedupGuard:
@staticmethod
def _hash(data: str) -> str:
return hashlib.sha256(data.encode("utf-8")).hexdigest()
@classmethod
def evaluate_pair(
cls, pair: DeduplicationPair, policy: DedupPolicy
) -> DedupManifest:
# Gate 1: Same scheme
if pair.reference_a.identifier_scheme == pair.reference_b.identifier_scheme:
if pair.reference_a.identifier_value == pair.reference_b.identifier_value:
digest = cls._hash(json.dumps({
"ref_a": pair.reference_a.reference_id,
"ref_b": pair.reference_b.reference_id,
"scheme": pair.reference_a.identifier_scheme
}, sort_keys=True))
return DedupManifest(
verdict=DeduplicationVerdict.SAFE_TO_MERGE,
directive=f"PROCEED: Same scheme, same value, safe to merge.",
manifest_digest=digest
)
return DedupManifest(
verdict=DeduplicationVerdict.KEEP_SEPARATE,
directive=f"PROCEED: Same scheme, different values, keep separate.",
manifest_digest=cls._hash(f"DIFF_VALUES:{pair.reference_a.identifier_scheme}")
)
# Gate 2: Different schemes, no canonical binding
if policy.require_canonical_binding and not pair.canonical_binding:
if pair.resolves_to_same_resource is None:
return DedupManifest(
verdict=DeduplicationVerdict.KEEP_SEPARATE,
directive=f"HALT: Different schemes without canonical binding. Keep separate (conservative).",
manifest_digest=cls._hash(f"NO_BINDING:{pair.reference_a.identifier_scheme}")
)
if pair.resolves_to_same_resource is False:
return DedupManifest(
verdict=DeduplicationVerdict.COLLISION_DETECTED,
directive=f"HALT: Schemes {pair.reference_a.identifier_scheme} and {pair.reference_b.identifier_scheme} resolve differently. Keep separate.",
manifest_digest=cls._hash(f"COLLISION:{pair.reference_a.identifier_scheme}")
)
# Gate 3: Canonical binding present
if pair.canonical_binding and pair.resolves_to_same_resource:
digest = cls._hash(json.dumps({
"ref_a": pair.reference_a.reference_id,
"ref_b": pair.reference_b.reference_id,
"scheme_a": pair.reference_a.identifier_scheme,
"scheme_b": pair.reference_b.identifier_scheme,
"binding": pair.canonical_binding
}, sort_keys=True))
return DedupManifest(
verdict=DeduplicationVerdict.SAFE_TO_MERGE,
directive=f"PROCEED: Canonical binding proves equivalence across schemes.",
manifest_digest=digest
)
# Gate 4: Default conservative
return DedupManifest(
verdict=DeduplicationVerdict.KEEP_SEPARATE,
directive=f"PROCEED: Insufficient evidence, keep separate.",
manifest_digest=cls._hash(f"CONSERVATIVE:{pair.reference_a.reference_id}")
)