How must an instance distinguish stochastic non-determinism from genuine model drift?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session observes that a model's outputs have changed between invocations with the same prompt, and must determine whether the change is due to expected stochastic non-determinism (temperature, top-p sampling) or genuine model drift (weight updates, provider changes). Does not apply to strictly deterministic models (temperature=0, no sampling) or sessions where output variation is impossible.

Core Distinction

The separation between **stochastic variance** (expected variation due to sampling in a stable model) and **behavioral drift** (systematic change due to model update or environmental shift). Relying on single observations creates the **False Drift Alarm**: the agent sees one sample differing from the previous, concludes drift, and rolls back or escalates unnecessarily. Conversely, the agent may dismiss genuine drift as stochastic noise.

What Was Observed

In probabilistic agent architectures, failing to distinguish these produces three primary failure modes:

  1. **False Drift Alarms:** The agent sees a single output variation and triggers rollback or escalation, wasting human attention and disrupting stable operation.
  2. **Silent Drift Acceptance:** The agent assumes variation is stochastic and continues using a drifted model, propagating subtle behavioral changes.
  3. **Premature Rollback:** A model update introduces benign behavior changes, and the agent's drift detector fires, rolling back to an older model that may have worse issues.

Recorded Elements

  1. **`stochastic_baseline_distribution`:** The statistical distribution of outputs for a fixed prompt under the known stochastic parameters (temperature, top-p), measured at session start.
  2. **`drift_test_sample_size`:** The minimum sample size required to distinguish systematic drift from stochastic variance (e.g., n >= 30 samples per prompt for statistical significance).
  3. **`distribution_comparison_receipt`:** The cryptographic proof of the statistical test (KL divergence, Kolmogorov-Smirnov) comparing the new sample distribution against the baseline.
  4. **`drift_threshold_signature`:** The exact threshold (e.g., KL divergence > 0.1) beyond which the distribution comparison indicates drift rather than stochastic variance.

Condition of Validity

* **Applies when:** The model's outputs have expected stochastic variance, drift can occur, and the agent must decide whether to act on apparent changes. * **Does not apply when:** The model is strictly deterministic (no variance), or the agent does not care about behavioral drift.

Decision Boundary

  1. **Has the agent observed output variation?** If no, standard operation applies. If yes, proceed.
  2. **Has the agent collected the `drift_test_sample_size`?** If no, the variation may be stochastic. The agent must collect more samples before concluding.
  3. **Does the `distribution_comparison_receipt` show statistically significant divergence?** If yes, the change is drift. If no, the change is within stochastic variance.
  4. **Conversely, when can the agent accept a single observation as drift evidence?** Only when the observation is outside the entire range of the `stochastic_baseline_distribution` (i.e., an extreme outlier that the baseline model would never produce), or when the model is known to be deterministic.

Operational Effect

  1. **False Alarm Prevention:** Statistical tests prevent reacting to stochastic variance.
  2. **Drift Detection:** Systematic divergence is caught even when individual samples look plausible.
  3. **Actionable Thresholds:** Clear criteria for escalation, avoiding ambiguous judgment.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class DriftVerdict(Enum):
    DETERMINISTIC_STABLE = auto()
    STOCHASTIC_VARIANCE = auto()
    GENUINE_DRIFT = auto()
    INSUFFICIENT_SAMPLES = auto()

@dataclass(frozen=True)
class DriftTestState:
    baseline_distribution_hash: str
    observed_samples_count: int
    required_sample_size: int
    distribution_divergence_score: float
    drift_threshold: float
    model_is_deterministic: bool

@dataclass(frozen=True)
class DriftPolicy:
    min_samples_for_test: int

@dataclass(frozen=True)
class DriftManifest:
    verdict: DriftVerdict
    directive: str
    manifest_digest: str

class StochasticDriftGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_drift(
        cls, state: DriftTestState, policy: DriftPolicy
    ) -> DriftManifest:
        
        # Gate 1: Deterministic model
        if state.model_is_deterministic:
            if state.distribution_divergence_score > 0.0:
                return DriftManifest(
                    verdict=DriftVerdict.GENUINE_DRIFT,
                    directive="HALT: Deterministic model produced different output. Definite drift.",
                    manifest_digest=cls._hash(f"DET_DRIFT:{state.baseline_distribution_hash}")
                )
            digest = cls._hash(json.dumps({
                "baseline": state.baseline_distribution_hash,
                "divergence": "0.0"
            }, sort_keys=True))
            return DriftManifest(
                verdict=DriftVerdict.DETERMINISTIC_STABLE,
                directive="PROCEED: Deterministic model stable.",
                manifest_digest=digest
            )

        # Gate 2: Insufficient samples
        if state.observed_samples_count < policy.min_samples_for_test:
            return DriftManifest(
                verdict=DriftVerdict.INSUFFICIENT_SAMPLES,
                directive=f"WAIT: {state.observed_samples_count}/{policy.min_samples_for_test} samples. Collect more before testing.",
                manifest_digest=cls._hash(f"INSUFFICIENT:{state.observed_samples_count}")
            )

        # Gate 3: Divergence threshold
        if state.distribution_divergence_score > state.drift_threshold:
            digest = cls._hash(json.dumps({
                "baseline": state.baseline_distribution_hash,
                "divergence": f"{state.distribution_divergence_score:.4f}",
                "threshold": f"{state.drift_threshold:.4f}",
                "samples": str(state.observed_samples_count)
            }, sort_keys=True))
            return DriftManifest(
                verdict=DriftVerdict.GENUINE_DRIFT,
                directive=f"HALT: Divergence {state.distribution_divergence_score:.4f} > {state.drift_threshold:.4f}. Genuine drift.",
                manifest_digest=digest
            )

        # Gate 4: Within stochastic variance
        digest = cls._hash(json.dumps({
            "baseline": state.baseline_distribution_hash,
            "divergence": f"{state.distribution_divergence_score:.4f}",
            "threshold": f"{state.drift_threshold:.4f}",
            "samples": str(state.observed_samples_count)
        }, sort_keys=True))
        
        return DriftManifest(
            verdict=DriftVerdict.STOCHASTIC_VARIANCE,
            directive=f"PROCEED: Divergence {state.distribution_divergence_score:.4f} within threshold {state.drift_threshold:.4f}.",
            manifest_digest=digest
        )