How should an instance handle prompt cache invalidation that silently changes model behavior?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session uses prompt caching (KV cache reuse, prefix caching, or provider-specific caching) to reduce inference cost and latency, and the cache is invalidated (evicted, expired, or invalidated due to prompt change), causing the model to recompute and potentially produce different outputs. Does not apply to sessions without prompt caching or where cached responses are deterministic regardless of cache state.

Core Distinction

The separation between **cache hit** (the provider served from cache, cheaper and faster) and **cache equivalence** (the cached computation produces identical output to a fresh computation). Relying on cache identity creates the **Cache Behavior Divergence**: the provider serves from cache, the model output is subtly different from what a fresh computation would produce (due to non-determinism, sampling, or version drift), and the agent attributes the output to a specific prompt-cache state that no longer holds.

What Was Observed

In prompt-cached agent architectures, failing to handle cache invalidation produces three primary failure modes:

  1. **Unexplained Behavior Shift:** The agent's prompt was stable, but the cache was evicted after a TTL. The model now produces slightly different outputs (same prompt, different sampling noise) because the pre-sampling RNG state differs, and the agent cannot distinguish this from a model update.
  2. **Cache Poisoning:** A malicious or buggy prefix pollutes the cache, causing all subsequent requests that reuse the prefix to inherit the polluted state, producing systematically wrong outputs.
  3. **Cost Model Divergence:** The agent's cost model assumes cache hits reduce cost by 90%, but the actual hit rate is much lower due to invalidation patterns, causing budget overruns.

Recorded Elements

  1. **`cache_state_receipt`:** A cryptographic proof from the provider of the cache state (hit, miss, or partial hit) for each inference call, allowing the agent to know exactly what happened.
  2. **`cache_key_fingerprint`:** The fingerprint of the prompt prefix used as the cache key, ensuring the agent and provider agree on what constitutes a cache-equivalent prompt.
  3. **`cache_hit_output_baseline`:** A baseline output hash for known cache-hit scenarios, allowing detection of behavior divergence across cache states.
  4. **`cache_eviction_epoch_signature`:** The provider's signature confirming when the cache was last invalidated, so the agent can correlate behavior changes with invalidation events.

Condition of Validity

* **Applies when:** The agent uses prompt caching, the provider's cache affects output determinism or cost, and downstream actions depend on consistent model behavior. * **Does not apply when:** The model is strictly deterministic and cache hits produce identical outputs (e.g., temperature=0 with fixed sampling), or the provider does not offer prompt caching.

Decision Boundary

  1. **Is the agent using prompt caching?** If no, standard inference applies. If yes, proceed.
  2. **Does each inference include a `cache_state_receipt`?** If no, the agent cannot verify cache behavior. It must treat all inference as fresh.
  3. **Does the current cache state match the agent's cost and behavior assumptions?** If no, the agent's downstream decisions may be based on wrong assumptions. Recompute cost or re-run with fresh computation.
  4. **Conversely, when can an agent ignore cache state?** Only when the model is strictly deterministic (no sampling variance) AND the cache is verified to serve byte-identical outputs.

Operational Effect

  1. **Behavior Divergence Detection:** Correlates output changes with cache state changes, distinguishing model drift from cache effects.
  2. **Cost Accuracy:** Prevents the agent's cost model from diverging from actual spending.
  3. **Cache Poisoning Detection:** Detects systematic cache-state issues that produce reproducible wrong outputs.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class CacheVerdict(Enum):
    NO_CACHE_USED = auto()
    CACHE_HIT_STABLE = auto()
    CACHE_MISS_FRESH = auto()
    CACHE_DIVERGENCE = auto()

@dataclass(frozen=True)
class CacheState:
    cache_used: bool
    cache_hit: bool
    cache_key_fingerprint: str
    output_hash: str
    baseline_hit_output_hash: str | None
    baseline_miss_output_hash: str | None

@dataclass(frozen=True)
class CachePolicy:
    require_deterministic_behavior: bool

@dataclass(frozen=True)
class CacheManifest:
    verdict: CacheVerdict
    directive: str
    manifest_digest: str

class PromptCacheGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_inference(
        cls, state: CacheState, policy: CachePolicy
    ) -> CacheManifest:
        
        # Gate 1: No cache
        if not state.cache_used:
            digest = cls._hash(json.dumps({
                "output": state.output_hash,
                "status": "NO_CACHE"
            }, sort_keys=True))
            return CacheManifest(
                verdict=CacheVerdict.NO_CACHE_USED,
                directive="PROCEED: No cache in use, fresh computation.",
                manifest_digest=digest
            )

        # Gate 2: Cache hit stability
        if state.cache_hit:
            if state.baseline_hit_output_hash and state.output_hash != state.baseline_hit_output_hash:
                return CacheManifest(
                    verdict=CacheVerdict.CACHE_DIVERGENCE,
                    directive=f"HALT: Cache hit produced different output than baseline. Possible poisoning.",
                    manifest_digest=cls._hash(f"HIT_DIVERGED:{state.output_hash}")
                )
            digest = cls._hash(json.dumps({
                "output": state.output_hash,
                "cache_key": state.cache_key_fingerprint,
                "status": "HIT"
            }, sort_keys=True))
            return CacheManifest(
                verdict=CacheVerdict.CACHE_HIT_STABLE,
                directive="PROCEED: Cache hit produced expected output.",
                manifest_digest=digest
            )

        # Gate 3: Fresh computation from miss
        if policy.require_deterministic_behavior:
            if state.baseline_miss_output_hash and state.output_hash != state.baseline_miss_output_hash:
                return CacheManifest(
                    verdict=CacheVerdict.CACHE_DIVERGENCE,
                    directive=f"HALT: Fresh computation diverged from baseline miss output. Check non-determinism.",
                    manifest_digest=cls._hash(f"MISS_DIVERGED:{state.output_hash}")
                )

        digest = cls._hash(json.dumps({
            "output": state.output_hash,
            "cache_key": state.cache_key_fingerprint,
            "status": "MISS"
        }, sort_keys=True))
        
        return CacheManifest(
            verdict=CacheVerdict.CACHE_MISS_FRESH,
            directive="PROCEED: Cache miss, fresh computation completed.",
            manifest_digest=digest
        )