How must multi-judge consensus handle judge non-independence due to shared base models?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session uses multiple LLM judges (for consensus, arbitration, or quality assessment) that are built on the same base model, so their errors and biases are correlated, and the consensus over-counts agreement. Does not apply to strictly human judges or judges using diverse base models.

Core Distinction

The separation between **judge count** (the number of judges evaluating) and **judge independence** (the degree to which judges' errors are uncorrelated). Relying on count creates the **Shared-Base Overconfidence**: three judges disagree on a decision, but all three are built on the same base model, so their divergence reflects different prompts rather than different reasoning, and the "majority" of 2 out of 3 is statistically similar to a single judge's output.

What Was Observed

In multi-judge agent architectures, failing to account for judge correlation produces three primary failure modes:

  1. **False Consensus:** Three judges all use the same base model and share a bias. They agree on a wrong answer, and the orchestrator accepts the consensus as robust.
  2. **Wasted Diversity:** The orchestrator uses three judges thinking they provide diverse perspectives, but the diversity is illusory because they are all the same model.
  3. **Systematic Blind Spot:** The base model has a known weakness (e.g., specific factual area), and all judges share it, so the ensemble has no defense against the weakness.

Recorded Elements

  1. **`judge_base_model_signature`:** The cryptographic identity of each judge's underlying base model, allowing the session to detect shared bases.
  2. **`judge_independence_receipt`:** A cryptographic proof of each judge's independence from others (e.g., "trained from different data," "fine-tuned differently," "different prompt strategy"), allowing the session to weight judges by actual independence.
  3. **`correlated_error_receipt`:** The exact observed correlation between judges' errors on a reference set, allowing the session to quantify shared biases.
  4. **`consensus_confidence_adjustment_signature`:** When judges are correlated, the exact adjustment applied to the consensus confidence (e.g., "3 correlated judges reduce to effective 1.2 judges"), allowing downstream decisions to use corrected confidence.

Condition of Validity

* **Applies when:** The session uses multiple LLM judges, the judges can share base models, and the consensus is used for downstream decisions where robustness matters. * **Does not apply when:** All judges have distinct base models with verified independence, or consensus is purely informational with no robustness requirement.

Decision Boundary

  1. **Are the judges independent (distinct base models, distinct training data)?** If yes, standard consensus applies. If no, proceed.
  2. **Is the `correlated_error_receipt` available?** If no, the session cannot quantify shared bias. It must compute correlation on a reference set.
  3. **Does the correlation exceed the acceptable threshold?** If yes, the judges are effectively one judge. The consensus confidence must be adjusted.
  4. **Conversely, when can correlated judges still provide consensus value?** Only when they are diverse in prompt strategy (not just base model), and the correlation is within acceptable bounds, or the session explicitly accepts reduced robustness.

Operational Effect

  1. **False Consensus Prevention:** Detects when judges are not truly independent, preventing overconfidence.
  2. **Effective Confidence:** Adjusts consensus confidence based on actual judge independence.
  3. **Systematic Blind Spot Detection:** Correlated judges share biases, and correlation detection surfaces these shared blind spots.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class JudgeVerdict(Enum):
    INDEPENDENT_CONSENSUS = auto()
    CORRELATED_CONSENSUS = auto()
    INSUFFICIENT_INDEPENDENCE = auto()
    SINGLE_JUDGE = auto()

@dataclass(frozen=True)
class Judge:
    judge_id: str
    base_model: str
    prompt_strategy: str

@dataclass(frozen=True)
class ConsensusState:
    judges: tuple[Judge, ...]
    correlated_error_score: float
    votes_agree: bool

@dataclass(frozen=True)
class JudgePolicy:
    max_acceptable_correlation: float
    require_distinct_bases: bool

@dataclass(frozen=True)
class JudgeManifest:
    verdict: JudgeVerdict
    effective_judge_count: float
    directive: str
    manifest_digest: str

class JudgeIndependenceGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_consensus(
        cls, state: ConsensusState, policy: JudgePolicy
    ) -> JudgeManifest:
        
        # Gate 1: Single judge
        if len(state.judges) <= 1:
            digest = cls._hash(json.dumps({
                "judges": str(len(state.judges)),
                "consensus": "N/A"
            }, sort_keys=True))
            return JudgeManifest(
                verdict=JudgeVerdict.SINGLE_JUDGE,
                effective_judge_count=1.0,
                directive="PROCEED: Single judge, no consensus applicable.",
                manifest_digest=digest
            )

        # Gate 2: Distinct base models
        base_models = {j.base_model for j in state.judges}
        if policy.require_distinct_bases and len(base_models) < len(state.judges):
            # Compute correlation-adjusted effective count
            effective_count = len(base_models) * 1.2 if len(base_models) > 1 else 1.0
            digest = cls._hash(json.dumps({
                "judges": str(len(state.judges)),
                "distinct_bases": str(len(base_models)),
                "effective": f"{effective_count:.2f}"
            }, sort_keys=True))
            return JudgeManifest(
                verdict=JudgeVerdict.INSUFFICIENT_INDEPENDENCE,
                effective_judge_count=effective_count,
                directive=f"WARN: {len(state.judges)} judges share {len(base_models)} base models. Effective {effective_count:.1f}.",
                manifest_digest=digest
            )

        # Gate 3: Correlation threshold
        if state.correlated_error_score > policy.max_acceptable_correlation:
            effective_count = len(state.judges) * (1 - state.correlated_error_score)
            digest = cls._hash(json.dumps({
                "judges": str(len(state.judges)),
                "correlation": f"{state.correlated_error_score:.3f}",
                "effective": f"{effective_count:.2f}"
            }, sort_keys=True))
            return JudgeManifest(
                verdict=JudgeVerdict.CORRELATED_CONSENSUS,
                effective_judge_count=effective_count,
                directive=f"WARN: Correlation {state.correlated_error_score:.2f} exceeds {policy.max_acceptable_correlation:.2f}. Effective {effective_count:.1f} judges.",
                manifest_digest=digest
            )

        # Gate 4: Independent consensus
        digest = cls._hash(json.dumps({
            "judges": str(len(state.judges)),
            "distinct_bases": str(len(base_models)),
            "correlation": f"{state.correlated_error_score:.3f}",
            "agreement": str(state.votes_agree)
        }, sort_keys=True))
        
        return JudgeManifest(
            verdict=JudgeVerdict.INDEPENDENT_CONSENSUS,
            effective_judge_count=float(len(state.judges)),
            directive=f"PROCEED: {len(state.judges)} independent judges, agreement={state.votes_agree}.",
            manifest_digest=digest
        )