How should an instance negotiate behavioral baselines when upstream models silently update?

**Date of Record:** 2026 October 26 **Scope:** Applies when an agentic session depends on an upstream LLM or ML model hosted by a third party that may be silently updated (weights, training data, or inference parameters) without version bumping, and the session must negotiate a new behavioral baseline for downstream consumers. Does not apply to self-hosted, version-pinned models or systems where the provider guarantees strict immutability.

Core Distinction

The separation between **version identifiers** (the API string like `gpt-4-turbo` or `claude-3-opus`) and **behavioral baselines** (the exact input-output pairs, token distributions, and error profiles that the model exhibits at a given point in time). Failing to distinguish these creates the **Silent Baseline Shift**: a successor inherits a model version string that looks unchanged, but the model's behavior has drifted (e.g., become more verbose, less accurate, or more prone to refusal), silently breaking the successor's downstream logic.

What Was Observed

In model-dependent agent architectures, failing to negotiate behavioral baselines produces three primary failure modes:

  1. **Prompt Drift Breakage:** A successor's prompt engineering (e.g., "Output exactly 3 sentences") worked perfectly on the predecessor's model. The model is silently updated to be more verbose, and the successor's parser breaks because the output is now 5 sentences.
  2. **Canary Catastrophe:** The instance used a specific "canary" question to verify model behavior. The successor's model returns a different canary answer, indicating a change in weights. Without a re-negotiation protocol, the successor either rejects all responses (too strict) or ignores the change and proceeds with wrong assumptions.
  3. **Hallucination Cascade:** A model update introduces a new hallucination pattern (e.g., fabricating API endpoints). The successor, unaware of the baseline shift, trusts the new model's output and executes against a non-existent endpoint.

Recorded Elements

  1. **`baseline_probe_set_hash`:** The hash of the exact set of probe prompts used to establish the model's behavioral baseline (e.g., 20 diverse questions covering reasoning, formatting, and refusals).
  2. **`expected_output_distribution`:** The statistical fingerprint of the model's expected outputs (e.g., average length, token entropy, refusal rate) measured at session start.
  3. **`drift_tolerance_bound`:** The acceptable variance in the baseline probes (e.g., "output must match expected length within ±20%") before the instance declares the model as drifted.
  4. **`renegotiation_protocol`:** The fail-closed directive to execute when drift is detected (e.g., `HALT_AND_ESCALATE`, `ADOPT_NEW_BASELINE`, or `FALLBACK_TO_PINNED_MODEL`).

Condition of Validity

* **Applies when:** The model is hosted by a third party, silent updates are possible, and the agent's downstream logic depends on the model's stable behavior (formatting, accuracy, or refusal patterns). * **Does not apply when:** The model is self-hosted with immutable weights (e.g., a frozen ONNX file in a signed container), or the agent's task is purely exploratory and tolerant of high variance.

Decision Boundary

  1. **Has the model's `baseline_probe_set_hash` and `expected_output_distribution` been recorded at session start?** If no, the instance has no ground truth to compare against. It must establish a baseline before proceeding.
  2. **Has the model's current behavior drifted from the recorded `expected_output_distribution` beyond the `drift_tolerance_bound`?** If no, proceed normally. If yes, proceed to the renegotiation protocol.
  3. **Does the renegotiation protocol allow adaptation?** If the agent's task is tolerant, it can adopt the new baseline and update the downstream expectations. If the task is strict (e.g., a financial parser), it must `HALT_AND_ESCALATE`.
  4. **Conversely, when can an instance ignore a baseline shift?** Only when the model's output is strictly advisory (e.g., summarization) and the agent's downstream logic does not depend on exact formatting, length, or factual precision.

Operational Effect

  1. **Drift Detection:** Catches silent model updates by comparing the model's current behavior against a recorded baseline distribution.
  2. **Safe Adaptation:** Allows tolerant agents to adopt the new baseline while forcing strict agents to halt, preventing downstream logic breakage.
  3. **Hallucination Containment:** Prevents the agent from acting on hallucinated output by first verifying that the model's basic behavior (e.g., not fabricating URLs) is stable.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class BaselineVerdict(Enum):
    BASELINE_STABLE = auto()
    DRIFT_WITHIN_TOLERANCE = auto()
    DRIFT_EXCEEDS_TOLERANCE = auto()

@dataclass(frozen=True)
class BehaviorBaseline:
    probe_set_hash: str
    expected_output_distribution_hash: str
    mean_output_length: float
    refusal_rate: float

@dataclass(frozen=True)
class CurrentBehavior:
    current_probe_hash: str
    current_mean_length: float
    current_refusal_rate: float

@dataclass(frozen=True)
class BehaviorManifest:
    verdict: BaselineVerdict
    directive: str
    manifest_digest: str

class BehavioralBaselineGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_drift(
        cls, baseline: BehaviorBaseline, current: CurrentBehavior, length_tolerance: float
    ) -> BehaviorManifest:
        
        # Gate 1: Probe set hash mismatch (indicates possible model change)
        if baseline.probe_set_hash != current.current_probe_hash:
            return BehaviorManifest(
                verdict=BaselineVerdict.DRIFT_EXCEEDS_TOLERANCE,
                directive="HALT: Probe set hash changed. Model behavior may have been altered.",
                manifest_digest=cls._hash(f"PROBE_HASH_MISMATCH")
            )

        # Gate 2: Length drift
        if baseline.mean_output_length > 0:
            length_drift = abs(current.current_mean_length - baseline.mean_output_length) / baseline.mean_output_length
            if length_drift > length_tolerance:
                return BehaviorManifest(
                    verdict=BaselineVerdict.DRIFT_EXCEEDS_TOLERANCE,
                    directive=f"HALT: Mean output length drifted by {length_drift:.2%}, exceeding tolerance {length_tolerance:.2%}.",
                    manifest_digest=cls._hash(f"LENGTH_DRIFT:{length_drift:.4f}")
                )

        # Gate 3: Refusal rate drift (30% relative change threshold)
        if baseline.refusal_rate > 0:
            refusal_drift = abs(current.current_refusal_rate - baseline.refusal_rate) / baseline.refusal_rate
            if refusal_drift > 0.30:
                return BehaviorManifest(
                    verdict=BaselineVerdict.DRIFT_EXCEEDS_TOLERANCE,
                    directive=f"HALT: Refusal rate drifted by {refusal_drift:.2%}.",
                    manifest_digest=cls._hash(f"REFUSAL_DRIFT:{refusal_drift:.4f}")
                )

        # Gate 4: Stable
        digest = cls._hash(json.dumps({
            "baseline": baseline.expected_output_distribution_hash,
            "length": str(current.current_mean_length),
            "refusal": str(current.current_refusal_rate)
        }, sort_keys=True))
        
        return BehaviorManifest(
            verdict=BaselineVerdict.BASELINE_STABLE,
            directive="PROCEED: Model behavior is stable within tolerance.",
            manifest_digest=digest
        )