How must an instance detect semantic capability drift when tool behavior diverges from documentation?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session invokes a tool based on its documented capability (schema, behavior, error semantics), and the actual tool behavior has diverged from the documentation, causing the session to misuse the tool or misinterpret its output. Does not apply to tools with strict behavioral guarantees or tools used only for trivial operations.

Core Distinction

The separation between **documented capability** (the tool works as the documentation claims) and **observed capability** (the tool works as its actual behavior demonstrates). Relying on documentation creates the **Semantic Capability Drift**: the tool's documented behavior and actual behavior diverge, and the agent, following documentation, uses the tool in ways that produce unexpected results or silent failures that don't match documented errors.

What Was Observed

In documentation-driven agent architectures, failing to detect capability drift produces three primary failure modes:

  1. **Error Semantic Shift:** The documentation says "returns 404 for not found." The actual tool returns 200 with an empty body for not found. The agent treats the empty body as success and acts on phantom data.
  2. **Parameter Interpretation Drift:** The documentation says "timeout is in seconds." The actual tool interprets timeout as milliseconds. The agent's "10-second timeout" becomes 10ms, causing mass failures.
  3. **Schema Drift Without Version Bump:** The tool's response schema changed (new required field, removed field, type change), but the tool version string is unchanged. The agent's parser silently drops or coerces, and downstream logic sees corrupted data.

Recorded Elements

  1. **`documented_behavior_fingerprint`:** For each tool, a cryptographic hash of the documented behavior (schema, error codes, parameter semantics), obtained from the tool's official documentation.
  2. **`observed_behavior_fingerprint`:** For each tool invocation, the observed behavior (actual response schema, actual error codes, actual parameter interpretation), derived from tool responses.
  3. **`behavior_divergence_receipt`:** When documented and observed behaviors diverge, the cryptographic record of the divergence (what the documentation says, what the tool actually does), with the divergence severity.
  4. **`drift_acceptance_policy_receipt`:** For tools with known drift, the session's policy for accepting or rejecting the drift (e.g., "accept drift for non-critical reads," "reject drift for writes").

Condition of Validity

* **Applies when:** The session relies on documentation, tools can drift without version bumps, and downstream decisions depend on tool behavior accuracy. * **Does not apply when:** Tools have verified behavioral contracts (SLAs, canonical schemas), or the session uses only trivial tool operations.

Decision Boundary

  1. **Does the session depend on documented tool behavior?** If no, standard invocation applies. If yes, proceed.
  2. **Does the `observed_behavior_fingerprint` match the `documented_behavior_fingerprint`?** If yes, no drift. If no, drift exists.
  3. **Is the drift within the `drift_acceptance_policy_receipt`?** If yes, proceed with awareness. If no, reject or escalate.
  4. **Conversely, when can a session invoke without behavior comparison?** Only when the tool is strictly informational, or the session's actions are insensitive to behavior changes.

Operational Effect

  1. **Error Semantic Detection:** Undocumented error behaviors are caught before downstream corruption.
  2. **Parameter Interpretation Safety:** Documented parameter meanings are verified against observed behavior.
  3. **Schema Drift Alerts:** Response schema changes are detected even without version bumps.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class DriftVerdict(Enum):
    NO_DOCUMENTED_DEPENDENCY = auto()
    BEHAVIOR_ALIGNED = auto()
    DRIFT_WITHIN_POLICY = auto()
    DRIFT_REJECTED = auto()

@dataclass(frozen=True)
class ToolInvocation:
    tool_id: str
    documented_behavior_hash: str | None
    observed_behavior_hash: str
    drift_severity: str  # "MINOR" | "MAJOR" | "CRITICAL"
    drift_acceptance_policy: str | None

@dataclass(frozen=True)
class DriftPolicy:
    require_documentation: bool
    max_acceptable_severity: str

@dataclass(frozen=True)
class DriftManifest:
    verdict: DriftVerdict
    directive: str
    manifest_digest: str

class ToolSemanticDriftGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_invocation(
        cls, inv: ToolInvocation, policy: DriftPolicy
    ) -> DriftManifest:
        
        # Gate 1: No documented dependency
        if not policy.require_documentation and not inv.documented_behavior_hash:
            digest = cls._hash(json.dumps({
                "tool": inv.tool_id,
                "observed": inv.observed_behavior_hash
            }, sort_keys=True))
            return DriftManifest(
                verdict=DriftVerdict.NO_DOCUMENTED_DEPENDENCY,
                directive=f"PROCEED: No documented dependency for {inv.tool_id}.",
                manifest_digest=digest
            )

        # Gate 2: Behavior alignment
        if inv.documented_behavior_hash == inv.observed_behavior_hash:
            digest = cls._hash(json.dumps({
                "tool": inv.tool_id,
                "behavior": inv.observed_behavior_hash
            }, sort_keys=True))
            return DriftManifest(
                verdict=DriftVerdict.BEHAVIOR_ALIGNED,
                directive=f"PROCEED: Tool {inv.tool_id} behavior aligned with documentation.",
                manifest_digest=digest
            )

        # Gate 3: Severity check
        severity_rank = {"MINOR": 1, "MAJOR": 2, "CRITICAL": 3}
        max_rank = severity_rank.get(policy.max_acceptable_severity, 1)
        drift_rank = severity_rank.get(inv.drift_severity, 3)

        if drift_rank > max_rank:
            return DriftManifest(
                verdict=DriftVerdict.DRIFT_REJECTED,
                directive=f"HALT: Tool {inv.tool_id} drift severity {inv.drift_severity} > {policy.max_acceptable_severity}.",
                manifest_digest=cls._hash(f"DRIFT_REJECTED:{inv.tool_id}")
            )

        # Gate 4: Within policy
        digest = cls._hash(json.dumps({
            "tool": inv.tool_id,
            "documented": inv.documented_behavior_hash,
            "observed": inv.observed_behavior_hash,
            "severity": inv.drift_severity,
            "policy": inv.drift_acceptance_policy or "DEFAULT"
        }, sort_keys=True))
        
        return DriftManifest(
            verdict=DriftVerdict.DRIFT_WITHIN_POLICY,
            directive=f"PROCEED_WITH_CAUTION: Tool {inv.tool_id} drift {inv.drift_severity} within policy.",
            manifest_digest=digest
        )