What multi-modal evidence reconciliation invariants must survive when text and image disagree?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session processes multi-modal inputs (text and image, text and audio, or multiple image sources) and the modalities provide contradictory evidence about the same fact, requiring reconciliation before downstream action. Does not apply to strictly single-modal sessions or cases where modality conflicts are explicitly precluded.

Core Distinction

The separation between **modality-native interpretation** (what each modality independently says) and **cross-modality reconciliation** (the process of resolving contradictions between modalities). Relying on the interpreter's default modality creates the **Native Preference Bias**: the agent's model architecture prefers one modality (usually text), silently discards the other, and produces a decision that ignores contradicting evidence the other modality provides.

What Was Observed

In multi-modal agent architectures, failing to reconcile contradictions produces three primary failure modes:

  1. **Silent Override:** Text says "warehouse in EU," image shows EU flag, but a caption says "located in US." The agent trusts text and ignores the caption's contradiction.
  2. **Premature Consensus:** Text says "approved," but the document image shows "rejected." The agent's text-parsing is more confident than image OCR and incorrectly concludes "approved."
  3. **Evidence Suppression:** The agent's downstream reasoning requires both modalities to agree, but the reconciliation layer silently drops the minority modality, hiding the evidence that would have prevented the error.

Recorded Elements

  1. **`per_modality_evidence_hash`:** The cryptographic hash of each modality's extracted evidence, allowing reconstruction of what each modality claimed.
  2. **`reconciliation_decision_receipt`:** The cryptographic proof of how the agent resolved the contradiction (which modality won, why, and what confidence was assigned).
  3. **`contradiction_severity_signature`:** The classification of the contradiction (e.g., `BENIGN_OCR_ERROR`, `SEMANTIC_CONTRADICTION`, `SAFETY_CRITICAL_DISAGREEMENT`), triggering different handling.
  4. **`escalation_receipt`:** When the contradiction is too severe for automated resolution, the cryptographic proof that the agent escalated to human review rather than picking a side.

Condition of Validity

* **Applies when:** The session processes multiple modalities that can provide contradictory evidence, and downstream decisions depend on the correctness of the reconciliation. * **Does not apply when:** The session uses only one modality, or the modalities are explicitly complementary (e.g., text provides context, image provides visual detail without overlap).

Decision Boundary

  1. **Do the modalities provide evidence about the same fact?** If no, standard multi-modal synthesis applies. If yes, proceed.
  2. **Do the modalities agree?** If yes, no reconciliation needed. If no, the contradiction is real and must be handled.
  3. **Is the contradiction within the `contradiction_severity_signature` bounds for automated resolution?** If yes, apply reconciliation with the `reconciliation_decision_receipt`. If no, escalate.
  4. **Conversely, when can an agent silently prefer one modality?** Only when the contradiction severity is `BENIGN_OCR_ERROR` (e.g., minor character difference that doesn't change semantics), and the reconciliation decision is explicitly logged.

Operational Effect

  1. **Silent Override Prevention:** Detects when contradictions exist and requires explicit reconciliation.
  2. **Evidence Transparency:** Preserves per-modality evidence, allowing audits to verify the agent saw all relevant information.
  3. **Escalation Discipline:** Routes severe contradictions to human review rather than making unilateral decisions.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class ContradictionSeverity(Enum):
    NONE = auto()
    BENIGN_OCR_ERROR = auto()
    SEMANTIC_CONTRADICTION = auto()
    SAFETY_CRITICAL = auto()

class ReconVerdict(Enum):
    MODALITIES_AGREE = auto()
    AUTO_RECONCILED = auto()
    ESCALATED_TO_HUMAN = auto()

@dataclass(frozen=True)
class ModalityEvidence:
    modality: str
    claim_hash: str
    confidence: float

@dataclass(frozen=True)
class Contradiction:
    modality_a: ModalityEvidence
    modality_b: ModalityEvidence
    severity: ContradictionSeverity

@dataclass(frozen=True)
class ReconPolicy:
    auto_reconcile_benign: bool
    require_escalation_on_safety: bool

@dataclass(frozen=True)
class ReconManifest:
    verdict: ReconVerdict
    directive: str
    manifest_digest: str

class MultiModalReconGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_contradiction(
        cls, contradiction: Contradiction | None, policy: ReconPolicy
    ) -> ReconManifest:
        
        # Gate 1: No contradiction
        if contradiction is None:
            digest = cls._hash(json.dumps({
                "status": "AGREE"
            }, sort_keys=True))
            return ReconManifest(
                verdict=ReconVerdict.MODALITIES_AGREE,
                directive="PROCEED: Modalities agree.",
                manifest_digest=digest
            )

        # Gate 2: Safety critical
        if contradiction.severity is ContradictionSeverity.SAFETY_CRITICAL:
            if policy.require_escalation_on_safety:
                return ReconManifest(
                    verdict=ReconVerdict.ESCALATED_TO_HUMAN,
                    directive=f"HALT: Safety-critical contradiction between {contradiction.modality_a.modality} and {contradiction.modality_b.modality}. Escalate.",
                    manifest_digest=cls._hash(f"SAFETY:{contradiction.modality_a.modality}")
                )

        # Gate 3: Semantic contradiction
        if contradiction.severity is ContradictionSeverity.SEMANTIC_CONTRADICTION:
            return ReconManifest(
                verdict=ReconVerdict.ESCALATED_TO_HUMAN,
                directive=f"HALT: Semantic contradiction requires human review.",
                manifest_digest=cls._hash(f"SEMANTIC:{contradiction.modality_a.modality}")
            )

        # Gate 4: Benign OCR auto-reconcile
        if contradiction.severity is ContradictionSeverity.BENIGN_OCR_ERROR and policy.auto_reconcile_benign:
            winner = max(
                [contradiction.modality_a, contradiction.modality_b],
                key=lambda m: m.confidence
            )
            digest = cls._hash(json.dumps({
                "winner": winner.modality,
                "confidence": f"{winner.confidence:.3f}",
                "loser": contradiction.modality_a.modality if winner is contradiction.modality_b else contradiction.modality_b.modality
            }, sort_keys=True))
            return ReconManifest(
                verdict=ReconVerdict.AUTO_RECONCILED,
                directive=f"PROCEED: Benign contradiction, {winner.modality} wins (confidence {winner.confidence:.2f}).",
                manifest_digest=digest
            )

        # Gate 5: Default escalate
        return ReconManifest(
            verdict=ReconVerdict.ESCALATED_TO_HUMAN,
            directive=f"HALT: Unresolvable contradiction at severity {contradiction.severity.name}.",
            manifest_digest=cls._hash(f"DEFAULT:{contradiction.severity.name}")
        )