What invariants prevent confirmation-biased retrieval from amplifying initial errors?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session performs iterative retrieval (search, RAG, or memory lookup) where each retrieval is influenced by accumulated context, and the retrieval system's tendency to return results consistent with the existing context can amplify initial errors into confident wrong conclusions. Does not apply to strictly one-shot retrieval or retrieval systems that are verified unbiased.

Core Distinction

The separation between **retrieval relevance** (returned content matches the query) and **retrieval independence** (the returned content provides evidence independent of the accumulated context). Relying on relevance creates the **Confirmation Retrieval Trap**: an initial error seeds the context, subsequent retrievals return content consistent with the error, the session interprets the consistent content as corroboration, and the error becomes more confident even though no new independent evidence has been gathered.

What Was Observed

In iterative-retrieval agent architectures, failing to detect confirmation bias produces three primary failure modes:

  1. **Error Seeding Amplification:** The session retrieves a wrong initial claim. Subsequent queries, informed by the wrong claim, retrieve content that supports it. The session treats the support as new evidence.
  2. **Query Framing Bias:** The session's retrieval queries are phrased to confirm the current hypothesis. Retrieval results match the query's framing and appear to confirm the hypothesis, but this is because the query was framed to confirm.
  3. **Consensus Fabrication:** The session retrieves the same wrong source multiple times via different queries, and treats each retrieval as independent corroboration.

Recorded Elements

  1. **`retrieval-context-coupling-receipt`:** For each retrieval, the exact context terms that influenced the query, so query framing is visible.
  2. **`retrieval-source-diversity-signature`:** For each retrieval batch, the set of distinct sources returned, so consensus fabrication is detectable.
  3. **`adversarial-retrieval-probe-receipt`:** For critical conclusions, the session's attempt to retrieve disconfirming evidence (queries designed to falsify the current conclusion), so confirmation bias is tested.
  4. **`query-framing-neutrality-signature`:** For critical retrievals, the neutrality of the query (no presupposition, balanced framing), so query framing bias is bounded.

Condition of Validity

* **Applies when:** The session performs iterative retrieval, retrieval is influenced by context, and errors can be amplified. * **Does not apply when:** Retrieval is one-shot, retrieval is verified unbiased, or downstream decisions are insensitive to retrieval quality.

Decision Boundary

  1. **Is the retrieval iterative (influenced by accumulated context)?** If no, single-shot applies. If yes, proceed.
  2. **Is `adversarial-retrieval-probe-receipt` present for critical conclusions?** If yes, disconfirmation is tested. If no, confirmation bias is unbounded.
  3. **Does `retrieval-source-diversity-signature` show sufficient diversity?** If yes, consensus is independent. If no, consensus is fabricated.
  4. **Conversely, when can a session accept a conclusion without disconfirmation test?** Only when the conclusion is not critical, or the retrieval's diversity confirms genuine independent corroboration.

Operational Effect

  1. **Error Seeding Prevention:** Disconfirming retrieval tests break amplification loops.
  2. **Query Framing Integrity:** Neutral queries prevent confirmation-only retrieval.
  3. **Consensus Integrity:** Source diversity distinguishes independent corroboration from fabrication.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class RetrievalBiasVerdict(Enum):
    NO_ITERATIVE_RETRIEVAL = auto()
    INDEPENDENT_RETRIEVAL = auto()
    CONFIRMATION_BIAS_SUSPECTED = auto()
    CONSENSUS_FABRICATION_DETECTED = auto()

@dataclass(frozen=True)
class RetrievalRound:
    round_id: str
    query_neutral: bool
    distinct_sources: frozenset[str]
    disconfirming_probe_run: bool
    context_terms_used: frozenset[str]

@dataclass(frozen=True)
class RetrievalSession:
    session_id: str
    rounds: tuple[RetrievalRound, ...]
    is_iterative: bool

@dataclass(frozen=True)
class RetrievalBiasPolicy:
    min_source_diversity: int
    require_disconfirming_for_critical: bool
    is_critical: bool

@dataclass(frozen=True)
class RetrievalBiasManifest:
    verdict: RetrievalBiasVerdict
    directive: str
    manifest_digest: str

class ConfirmationBiasRetrievalGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_session(
        cls, session: RetrievalSession, policy: RetrievalBiasPolicy
    ) -> RetrievalBiasManifest:
        
        # Gate 1: Not iterative
        if not session.is_iterative or len(session.rounds) <= 1:
            digest = cls._hash(json.dumps({
                "session": session.session_id,
                "type": "SINGLE_SHOT"
            }, sort_keys=True))
            return RetrievalBiasManifest(
                verdict=RetrievalBiasVerdict.NO_ITERATIVE_RETRIEVAL,
                directive=f"PROCEED: Session {session.session_id} not iterative.",
                manifest_digest=digest
            )

        # Gate 2: Disconfirmation requirement
        if policy.is_critical and policy.require_disconfirming_for_critical:
            no_probe = [r.round_id for r in session.rounds if not r.disconfirming_probe_run]
            if len(no_probe) == len(session.rounds):
                return RetrievalBiasManifest(
                    verdict=RetrievalBiasVerdict.CONFIRMATION_BIAS_SUSPECTED,
                    directive=f"HALT: Critical retrieval {session.session_id} lacks disconfirming probes.",
                    manifest_digest=cls._hash(f"NO_DISCONFIRM:{session.session_id}")
                )

        # Gate 3: Source diversity
        all_sources: set[str] = set()
        for round_data in session.rounds:
            all_sources.update(round_data.distinct_sources)

        if len(all_sources) < policy.min_source_diversity:
            return RetrievalBiasManifest(
                verdict=RetrievalBiasVerdict.CONSENSUS_FABRICATION_DETECTED,
                directive=f"FLAG: {len(session.rounds)} rounds but only {len(all_sources)} distinct sources.",
                manifest_digest=cls._hash(f"FABRICATION:{session.session_id}")
            )

        # Gate 4: Query framing
        for round_data in session.rounds:
            if not round_data.query_neutral:
                return RetrievalBiasManifest(
                    verdict=RetrievalBiasVerdict.CONFIRMATION_BIAS_SUSPECTED,
                    directive=f"FLAG: Round {round_data.round_id} has non-neutral framing.",
                    manifest_digest=cls._hash(f"FRAMED:{round_data.round_id}")
                )

        # Gate 5: Independent
        digest = cls._hash(json.dumps({
            "session": session.session_id,
            "rounds": str(len(session.rounds)),
            "sources": tuple(sorted(all_sources))
        }, sort_keys=True))
        
        return RetrievalBiasManifest(
            verdict=RetrievalBiasVerdict.INDEPENDENT_RETRIEVAL,
            directive=f"PROCEED: {len(session.rounds)} rounds across {len(all_sources)} sources.",
            manifest_digest=digest
        )