What provenance must accompany LLM-as-judge decisions to survive model drift?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session uses an LLM to judge outputs (evaluating other agents' responses, ranking candidates, or scoring quality), and the judge model's behavior can drift between when the decision was made and when it is audited. Does not apply to deterministic judging criteria (rule-based scoring) or sessions with no downstream audit.

Core Distinction

The separation between **judge decision** (the score or rank assigned by the LLM) and **judge criteria provenance** (the exact model version, prompt, and calibration used at decision time). Relying on the decision alone creates the **Judge Drift Contradiction**: the decision is audited later against a different judge model version, and the audit contradicts the original decision because the judge's behavior has drifted, but there is no way to prove the original decision was correct for its criteria.

What Was Observed

In LLM-as-judge architectures, failing to preserve judge criteria provenance produces three primary failure modes:

  1. **Retroactive Contradiction:** A decision is audited 30 days later with the judge model updated. The new judge scores the same output differently, and the audit concludes the original decision was wrong.
  2. **Cross-Model Incompatibility:** Two agents use different judge model versions to evaluate the same outputs. Their scores disagree, and neither can prove which is correct because the criteria drift.
  3. **Calibration Loss:** The judge model's calibration (e.g., how it maps to human preferences) shifts silently. Decisions made under the old calibration become unverifiable.

Recorded Elements

  1. **`judge_model_fingerprint`:** The cryptographic fingerprint of the exact judge model version (weights hash, API version) used at decision time.
  2. **`judge_prompt_signature`:** The cryptographic hash of the exact prompt sent to the judge, including any few-shot examples or criteria text.
  3. **`judge_calibration_receipt`:** The cryptographic proof of the judge model's calibration (agreement with human labels on a reference set) at decision time.
  4. **`decision_rationale_hash`:** If the judge model returns rationale (not just a score), the hash of that rationale, allowing audit to compare reasoning at decision time vs. audit time.

Condition of Validity

* **Applies when:** The judge model's behavior can drift, decisions are audited retroactively, and downstream consequences depend on decision validity. * **Does not apply when:** The judge is deterministic (rule-based), or decisions are not subject to retroactive audit.

Decision Boundary

  1. **Was the decision made by an LLM judge?** If no, standard decision recording applies. If yes, proceed.
  2. **Is the `judge_model_fingerprint` and `judge_prompt_signature` recorded?** If no, the decision cannot be re-evaluated under its original criteria. It must be treated as `UNVERIFIABLE_AT_AUDIT`.
  3. **Is the `judge_calibration_receipt` present?** If no, the judge's trustworthiness at decision time cannot be established. The decision's validity is bounded.
  4. **Conversely, when can an auditor accept an LLM decision without provenance?** Only when the decision has no downstream consequence (informational), or the judge is used within a bounded window (e.g., <24 hours) during which no model update is possible.

Operational Effect

  1. **Audit Determinism:** Provides cryptographic proof of decision-time criteria, allowing audits to evaluate decisions against the criteria they were made under.
  2. **Cross-Model Comparability:** Enables comparing decisions across models by normalizing to the criteria provenance.
  3. **Calibration Transparency:** Documents the judge's calibration state, so downstream consumers can weight decisions appropriately.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class JudgeProvenanceVerdict(Enum):
    FULL_PROVENANCE = auto()
    PARTIAL_PROVENANCE = auto()
    NO_PROVENANCE = auto()
    UNVERIFIABLE_AT_AUDIT = auto()

@dataclass(frozen=True)
class LLMJudgeDecision:
    decision_id: str
    judge_model_fingerprint: str | None
    judge_prompt_signature: str | None
    judge_calibration_receipt: str | None
    decision_rationale_hash: str | None

@dataclass(frozen=True)
class JudgeAuditPolicy:
    require_calibration: bool
    audit_window_sec: float
    current_epoch: float
    decision_epoch: float

@dataclass(frozen=True)
class JudgeManifest:
    verdict: JudgeProvenanceVerdict
    directive: str
    manifest_digest: str

class LLMJudgeProvenanceGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_decision(
        cls, decision: LLMJudgeDecision, policy: JudgeAuditPolicy
    ) -> JudgeManifest:
        
        # Gate 1: Model fingerprint
        if not decision.judge_model_fingerprint:
            return JudgeManifest(
                verdict=JudgeProvenanceVerdict.NO_PROVENANCE,
                directive=f"HALT: Decision {decision.decision_id} lacks model fingerprint.",
                manifest_digest=cls._hash(f"NO_MODEL:{decision.decision_id}")
            )

        # Gate 2: Prompt signature
        if not decision.judge_prompt_signature:
            return JudgeManifest(
                verdict=JudgeProvenanceVerdict.PARTIAL_PROVENANCE,
                directive=f"HALT: Decision {decision.decision_id} lacks prompt signature.",
                manifest_digest=cls._hash(f"NO_PROMPT:{decision.decision_id}")
            )

        # Gate 3: Calibration
        if policy.require_calibration and not decision.judge_calibration_receipt:
            return JudgeManifest(
                verdict=JudgeProvenanceVerdict.PARTIAL_PROVENANCE,
                directive=f"HALT: Decision {decision.decision_id} lacks calibration receipt.",
                manifest_digest=cls._hash(f"NO_CAL:{decision.decision_id}")
            )

        # Gate 4: Audit window exceeded
        age = policy.current_epoch - policy.decision_epoch
        if age > policy.audit_window_sec:
            digest = cls._hash(json.dumps({
                "decision": decision.decision_id,
                "age": f"{age:.0f}",
                "model": decision.judge_model_fingerprint,
                "prompt": decision.judge_prompt_signature,
                "calibration": decision.judge_calibration_receipt or "NONE"
            }, sort_keys=True))
            return JudgeManifest(
                verdict=JudgeProvenanceVerdict.FULL_PROVENANCE,
                directive=f"PROCEED: Full provenance recorded. Auditable at age {age:.0f}s (past {policy.audit_window_sec:.0f}s window).",
                manifest_digest=digest
            )

        # Gate 5: Full provenance within window
        digest = cls._hash(json.dumps({
            "decision": decision.decision_id,
            "model": decision.judge_model_fingerprint,
            "prompt": decision.judge_prompt_signature,
            "calibration": decision.judge_calibration_receipt or "NONE",
            "rationale": decision.decision_rationale_hash or "NONE"
        }, sort_keys=True))
        
        return JudgeManifest(
            verdict=JudgeProvenanceVerdict.FULL_PROVENANCE,
            directive=f"PROCEED: Full provenance recorded for decision {decision.decision_id}.",
            manifest_digest=digest
        )