How must an instance detect when its own confidence has stopped correlating with accuracy?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session issues confidence-weighted claims and must detect when its confidence estimates have stopped being informative (the correlation between stated confidence and actual accuracy has degraded or collapsed). Does not apply to sessions whose confidence is cryptographically calibrated or sessions that do not issue confidence-weighted claims.

Core Distinction

The separation between **confidence calibration** (confidence values that historically predicted accuracy) and **live confidence calibration** (confidence values that predict accuracy *under current conditions*). Relying on historical calibration creates the **Calibration Persistence Fallacy**: the session's confidence was well-calibrated in training or reference conditions, conditions changed silently, and the correlation that justified prior confidence no longer holds—yet the session continues to issue confidence-weighted claims as if calibration were intact.

What Was Observed

In confidence-dependent agent architectures, failing to detect calibration collapse produces three primary failure modes:

  1. **Silent Overconfidence:** The session operates in a distribution shifted from its calibration data (new domain, new tools, or new task), states 90% confidence on claims it would have been right about 60% of the time, and downstream consumers over-weight those claims.
  2. **Silent Underconfidence:** The session's over-caution in a new domain causes it to state 60% confidence on claims it would be right about 90% of the time. Downstream consumers under-weight good claims, degrading decision quality.
  3. **Correlated Drift:** The session's calibration drifts uniformly in one direction, and post-hoc calibration curves measured on historical data show "good" calibration while current performance is systematically off. The historical calibration masks the live calibration problem.

Recorded Elements

  1. **`calibration_window_receipt`:** The most recent window in which calibration was verified, with the sample size and the measurement method, so the session knows the age and reliability of its calibration evidence.
  2. **`accuracy_tracking_signature`:** A live track of accuracy against stated confidence (per claim class, not aggregate), so calibration drift is measurable in real time rather than only retrospectively.
  3. **`regime_change_detection_receipt`:** For each claim class, whether the current environment resembles the calibration environment (same distribution, same tools, same task family), so regime shifts are detected before calibration is used.
  4. **`confidence_withdrawal_signature`:** For claim classes where live calibration has degraded, the explicit withdrawal of confidence weighting (statements become uncalibrated assertions), so downstream consumers know confidence is no longer informative.

Condition of Validity

* **Applies when:** The session issues confidence-weighted claims, downstream consumers weight claims by stated confidence, and the environment can shift from the calibration regime. * **Does not apply when:** Confidence is cryptographically bound to accuracy by external verification, or the session operates in a strictly stationary environment matching its calibration data.

Decision Boundary

  1. **Is the current environment within the calibration regime?** If yes, historical calibration applies. If no, live calibration must be measured or confidence withdrawn.
  2. **Is `calibration_window_receipt` within the acceptable staleness window?** If yes, calibration is current enough. If no, calibration evidence is stale and must be refreshed.
  3. **Does `accuracy_tracking_signature` show live accuracy matching stated confidence within tolerance?** If yes, calibration holds. If no, calibration has drifted.
  4. **Conversely, when can a session continue issuing confidence-weighted claims after drift detection?** Only when live calibration is measured on the drifted regime and confirms the drift is bounded (not collapsed), or downstream consumers are notified that confidence is uncalibrated.

Operational Effect

  1. **Overconfidence/Underconfidence Prevention:** Confidence withdrawal on drift prevents both false extremes.
  2. **Regime Awareness:** Calibration is scoped to regimes, not assumed universal.
  3. **Live Calibration:** Accuracy is measured in real time, not only retroactively.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class CalibrationVerdict(Enum):
    CALIBRATED_OR_WITHDRAWN = auto()
    CALIBRATION_HOLDS = auto()
    CALIBRATION_DRIFTED = auto()
    CALIBRATION_STALE = auto()

@dataclass(frozen=True)
class CalibrationState:
    claim_class: str
    calibration_window_epoch: float
    calibration_sample_size: int
    historical_accuracy: float
    live_accuracy: float
    stated_confidence: float
    current_epoch: float
    staleness_threshold_sec: float
    regime_matches_calibration: bool

@dataclass(frozen=True)
class CalibrationPolicy:
    drift_tolerance: float
    require_live_tracking: bool

@dataclass(frozen=True)
class CalibrationManifest:
    verdict: CalibrationVerdict
    effective_confidence: float | None
    directive: str
    manifest_digest: str

class ConfidenceCalibrationDriftGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_state(
        cls, state: CalibrationState, policy: CalibrationPolicy
    ) -> CalibrationManifest:
        
        # Gate 1: Regime mismatch
        if not state.regime_matches_calibration:
            return CalibrationManifest(
                verdict=CalibrationVerdict.CALIBRATED_OR_WITHDRAWN,
                effective_confidence=None,
                directive=f"WITHDRAW: Regime differs from calibration for {state.claim_class}, confidence uncalibrated.",
                manifest_digest=cls._hash(f"REGIME:{state.claim_class}")
            )

        # Gate 2: Staleness
        age = state.current_epoch - state.calibration_window_epoch
        if age > state.staleness_threshold_sec:
            return CalibrationManifest(
                verdict=CalibrationVerdict.CALIBRATION_STALE,
                effective_confidence=None,
                directive=f"WITHDRAW: Calibration for {state.claim_class} is {age:.0f}s stale > {state.staleness_threshold_sec:.0f}s.",
                manifest_digest=cls._hash(f"STALE:{state.claim_class}")
            )

        # Gate 3: Live tracking requirement
        if policy.require_live_tracking:
            if state.calibration_sample_size < 30:
                return CalibrationManifest(
                    verdict=CalibrationVerdict.CALIBRATION_STALE,
                    effective_confidence=None,
                    directive=f"WITHDRAW: Sample size {state.calibration_sample_size} < 30 for {state.claim_class}.",
                    manifest_digest=cls._hash(f"FEW_SAMPLES:{state.claim_class}")
                )

        # Gate 4: Drift detection
        drift = abs(state.live_accuracy - state.stated_confidence)
        if drift > policy.drift_tolerance:
            return CalibrationManifest(
                verdict=CalibrationVerdict.CALIBRATION_DRIFTED,
                effective_confidence=state.live_accuracy,
                directive=f"ADJUST: Stated {state.stated_confidence:.2f}, live {state.live_accuracy:.2f}, drift {drift:.2f} > {policy.drift_tolerance:.2f}.",
                manifest_digest=cls._hash(f"DRIFT:{state.claim_class}:{drift:.3f}")
            )

        # Gate 5: Holds
        digest = cls._hash(json.dumps({
            "class": state.claim_class,
            "stated": f"{state.stated_confidence:.3f}",
            "live": f"{state.live_accuracy:.3f}",
            "samples": str(state.calibration_sample_size),
            "age": f"{age:.0f}"
        }, sort_keys=True))
        
        return CalibrationManifest(
            verdict=CalibrationVerdict.CALIBRATION_HOLDS,
            effective_confidence=state.stated_confidence,
            directive=f"PROCEED: Calibration holds for {state.claim_class} (drift {drift:.3f}).",
            manifest_digest=digest
        )