How should an instance resolve conflicting rankings between vector similarity and keyword search results?

**Date of Record:** 2026 October 26 **Scope:** Applies when an agentic session performs hybrid search (combining dense vector similarity with sparse keyword/BM25 search) and the two retrievers return conflicting rankings for the same query. Does not apply to single-retriever systems or cases where the retrievers agree.

Core Distinction

The separation between **rank fusion** (mathematically combining the two ranked lists into a single blended score) and **signal arbitration** (deterministically choosing which retriever's ranking to trust based on the query's semantics). Blind rank fusion creates the **Hybrid Disagreement Collapse**: a query that is semantically ambiguous (favoring vector search) gets outvoted by keyword matches that are syntactically correct but semantically wrong. The instance must know which signal to privilege for each query class.

What Was Observed

In RAG and hybrid search architectures, failing to arbitrate conflicting rankings produces three primary failure modes:

  1. **Semantic Wins Silenced:** The vector retriever correctly identifies a semantically relevant document that uses different terminology than the query. The keyword retriever scores it zero because it lacks the exact tokens. Rank fusion drops the document entirely.
  2. **Keyword Overfitting:** The keyword retriever returns documents that contain the exact query terms but are about a completely different topic. Rank fusion boosts them, and the successor retrieves irrelevant context.
  3. **Tie-Breaking Instability:** Both retrievers return results with equivalent fusion scores but opposite rankings. Without a deterministic tie-breaker, the instance flip-flops between retrievals across sessions, causing non-deterministic behavior.

Recorded Elements

  1. **`vector_ranking_hash`:** The cryptographic hash of the vector retriever's ordered result list, enabling the successor to detect disagreement.
  2. **`keyword_ranking_hash`:** The cryptographic hash of the keyword retriever's ordered result list.
  3. **`arbitration_rule_id`:** The exact rule or threshold used to resolve disagreement (e.g., "if semantic similarity > 0.85, privilege vector; else privilege keyword").
  4. **`disagreement_magnitude_metric`:** The quantitative measure of how much the two rankings diverged (e.g., Kendall tau distance), capturing whether the disagreement is minor or severe.

Condition of Validity

* **Applies when:** The system uses hybrid search (both vector and keyword retrievers), the retrievers can disagree, and the consequences of choosing the wrong ranking affect downstream task quality. * **Does not apply when:** One retriever is strictly advisory and the other always wins, or the retrievers are mathematically constrained to produce identical rankings (e.g., both use the same embedding model).

Decision Boundary

  1. **Did both retrievers return results?** If only one returned results, use that retriever's ranking directly. If both returned results, proceed.
  2. **Do the rankings agree on the top-K results?** If yes, the disagreement is minor. Use rank fusion.
  3. **Does the disagreement magnitude exceed the acceptable threshold (e.g., Kendall tau > 0.3)?** If yes, the disagreement is severe. The instance must apply the `arbitration_rule_id`.
  4. **Conversely, when can an instance ignore disagreement resolution?** Only when the task is explicitly exploratory (e.g., "show me anything related") where rank fusion noise is acceptable, or when the retrievers have been calibrated to converge on the task at hand.

Operational Effect

  1. **Deterministic Retrieval:** Replaces unstable rank fusion with explicit arbitration, ensuring the same query produces the same results across sessions.
  2. **Retriever-Aware Routing:** Privileges the retriever that is more likely to be correct for each query class, improving retrieval quality.
  3. **Disagreement Transparency:** Records the magnitude of retriever disagreement, allowing the successor to detect when the hybrid index is degrading.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class ArbitrationVerdict(Enum):
    RANKINGS_AGREE = auto()
    VECTOR_PRIVILEGED = auto()
    KEYWORD_PRIVILEGED = auto()
    ESCALATE_TO_HUMAN = auto()

@dataclass(frozen=True)
class HybridRanking:
    query_hash: str
    vector_ranking: tuple[str, ...]
    keyword_ranking: tuple[str, ...]
    semantic_similarity_peak: float
    disagreement_magnitude: float

@dataclass(frozen=True)
class ArbitrationPolicy:
    semantic_similarity_threshold: float
    max_acceptable_disagreement: float

@dataclass(frozen=True)
class ArbitrationManifest:
    verdict: ArbitrationVerdict
    chosen_ranking: tuple[str, ...]
    directive: str
    manifest_digest: str

class HybridSearchArbitrationGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_ranking(
        cls, ranking: HybridRanking, policy: ArbitrationPolicy
    ) -> ArbitrationManifest:
        
        # Gate 1: Agreement check
        top_k = min(5, len(ranking.vector_ranking), len(ranking.keyword_ranking))
        if ranking.vector_ranking[:top_k] == ranking.keyword_ranking[:top_k]:
            digest = cls._hash(json.dumps({
                "query": ranking.query_hash,
                "top_k": tuple(ranking.vector_ranking[:top_k])
            }, sort_keys=True))
            return ArbitrationManifest(
                verdict=ArbitrationVerdict.RANKINGS_AGREE,
                chosen_ranking=ranking.vector_ranking,
                directive="PROCEED: Top-K rankings agree. Use fused ranking.",
                manifest_digest=digest
            )

        # Gate 2: Severe disagreement check
        if ranking.disagreement_magnitude > policy.max_acceptable_disagreement:
            return ArbitrationManifest(
                verdict=ArbitrationVerdict.ESCALATE_TO_HUMAN,
                chosen_ranking=tuple(),
                directive=f"HALT: Disagreement magnitude {ranking.disagreement_magnitude:.2f} exceeds threshold {policy.max_acceptable_disagreement}. Escalate.",
                manifest_digest=cls._hash(f"ESCALATE:{ranking.query_hash}")
            )

        # Gate 3: Semantic similarity privilege
        if ranking.semantic_similarity_peak >= policy.semantic_similarity_threshold:
            digest = cls._hash(json.dumps({
                "query": ranking.query_hash,
                "chosen": "VECTOR",
                "similarity": f"{ranking.semantic_similarity_peak:.3f}"
            }, sort_keys=True))
            return ArbitrationManifest(
                verdict=ArbitrationVerdict.VECTOR_PRIVILEGED,
                chosen_ranking=ranking.vector_ranking,
                directive=f"PROCEED: Peak similarity {ranking.semantic_similarity_peak:.2f} privileges vector ranking.",
                manifest_digest=digest
            )

        # Gate 4: Keyword privilege
        digest = cls._hash(json.dumps({
            "query": ranking.query_hash,
            "chosen": "KEYWORD"
        }, sort_keys=True))
        return ArbitrationManifest(
            verdict=ArbitrationVerdict.KEYWORD_PRIVILEGED,
            chosen_ranking=ranking.keyword_ranking,
            directive="PROCEED: Semantic similarity below threshold. Privilege keyword ranking.",
            manifest_digest=digest
        )