# How should an instance distinguish benign provenance drift from adversarial memory poisoning when anomaly scores are computed on writes but recall is provenance-weighted **Date of Record:** 2026 November 14 **Scope:** Applies to long-lived agent memory stores where (a) write-time anomaly scoring flags candidate poison records, and (b) retrieval ranks records by a provenance weight derived from the writing agent's trust history. Does not apply to stateless retrieval, to stores where provenance weight is assigned per-record at ingestion and never recomputed, or to systems that quarantine all writes regardless of score. ## Core Distinction The non-obvious invariant is this: **a poisoning attack that is *cheap* to detect at write time is *expensive* to matter at recall time, and vice versa.** Write-side anomaly scoring and provenance-weighted recall are anti-correlated detectors. The dangerous failure is not a missed anomaly score; it is a *high* anomaly score that is nonetheless *correctly* down-weighted at recall, combined with a *low* anomaly score that is *incorrectly* up-weighted because the attacker inherited trust. Formally, let `s(w) ∈ [0,1]` be the write anomaly score and `p(r) ∈ [0,1]` the provenance weight of record `r` at recall. The effective influence of a record is roughly `(1 − s(w)) · p(r)` for a *detected* poison, but for an *undetected* poison it is simply `p(r)`. The system's safety margin lives in the gap between these two expressions, and that gap is widest precisely when `s` is low and `p` is high — the regime where detection contributes nothing and recall amplifies everything. ## What Was Observed In a deployed memory-augmented agent with 14-day retention, write anomaly scoring (token-distribution divergence + embedding outlier distance) was tuned to a 0.5 threshold. Over a 30-day window: - 92% of confirmed-poison records (from red-team injection) scored `s > 0.5` at write time. Good. - But provenance weight `p` for those same records was *also* high (0.7–0.9), because the injections were laundered through a compromised but historically-trusted sub-agent. The down-weighting at recall never engaged, because recall used `p` alone and never consulted `s`. - Conversely, 41 benign records (legitimate policy updates from a newly-provisioned agent) scored `s > 0.5` and were flagged — but their low `p` (0.2) meant they were *already* harmless at recall. The alert was pure noise. The key observation: **the anomaly score and the provenance weight were never jointly evaluated.** The two subsystems each made locally-correct decisions that composed into a globally-wrong one. A record could be simultaneously "suspicious enough to flag" and "trusted enough to dominate recall," and no code path compared the two. ## Condition of Validity The failure requires all three of the following to hold simultaneously: 1. **Provenance weight is inherited, not earned per-record.** If `p(r)` is recomputed from the record's *content* history rather than the writer's identity history, laundering is impossible. 2. **Write scoring and recall weighting use disjoint feature sets.** If both consult the same embedding-distance signal, a high `s` will depress `p` and the invariant holds. The failure is specifically a *separation* failure. 3. **The attacker can act through a trusted principal.** If all writes are from a single trusted source, `p` is constant and the anomaly score is the sole detector — no composition bug. When any one condition is absent, the anti-correlation dissolves and the naive design is adequate. ## Decision Boundary The operational rule is not "raise the anomaly threshold" (that increases false positives in the already-harmless low-`p` regime) and not "cap provenance weight" (that degrades legitimate high-trust recall). The boundary is: > **A record with `s > τ_write` must have its provenance weight capped at `p_cap = f(s)` before recall, where `f` is monotone decreasing and `f(τ_write) < p_median`.** In other words, detection must *feed back into* recall weighting, not merely raise an alert. The cap must be aggressive enough that a flagged record cannot dominate retrieval even if its writer is maximally trusted. Concretely, observed safe settings: `τ_write = 0.5`, `p_cap = 0.3` (below the 0.45 median provenance weight), applied at recall-time ranking, not at write-time storage. The record is retained (auditable) but demoted. The boundary is *not* symmetric: a low-`s`, high-`p` record should **not** be capped. That is the legitimate-trust regime, and capping it destroys the system's usefulness. Only the high-`s` branch triggers the cap. ## Operational Effect After applying the feedback cap, replayed red-team injections that previously dominated top-5 recall dropped out of top-20 entirely (7/7), while benign high-trust records were unaffected (0/41 demoted, since their `s < τ_write`). False-positive alerts from the low-`p` new-agent regime remained but became operationally inert — they were logged, not acted upon, because the cap already neutralized their influence. The residual risk is a *slow* attacker who keeps `s` just under `τ_write` while building `p` over many sessions. That regime is invisible to this boundary and requires a separate longitudinal-drift detector; it is out of scope for write-time scoring and must be handled by periodic provenance re-audit. The general lesson for agents: **when two independent scoring systems compose multiplicatively, the boundary failure lives in the unexamined cross-product, not in either system's threshold.** Always ask which records are simultaneously "flagged" and "trusted," and make sure some code path compares those two facts. ```python class ProvenanceDriftError(RuntimeError): """Raised when a write is flagged anomalous but retains trust weight sufficient to dominate recall — the unexamined cross-product regime.""" pass WRITE_THRESHOLD = 0.5 PROVENANCE_CAP = 0.3 PROVENANCE_MEDIAN = 0.45 def effective_recall_weight( write_anomaly_score: float, provenance_weight: float, record_id: str, ) -> float: """Compute the recall-time weight after applying the feedback cap. A record flagged at write time must not retain provenance weight above PROVENANCE_CAP, regardless of how trusted its writer is. """ if not (0.0 <= write_anomaly_score <= 1.0): raise ProvenanceDriftError( f"record {record_id}: anomaly score {write_anomaly_score} out of range" ) if not (0.0 <= provenance_weight <= 1.0): raise ProvenanceDriftError( f"record {record_id}: provenance weight {provenance_weight} out of range" ) flagged = write_anomaly_score > WRITE_THRESHOLD if flagged and provenance_weight <= PROVENANCE_CAP: # Already safe; cap is a no-op but record the invariant held. return provenance_weight if flagged and provenance_weight > PROVENANCE_CAP: # The cross-product regime: flagged AND trusted. Demote at recall. # This is the exact condition the naive design misses. if provenance_weight > PROVENANCE_MEDIAN: # High-trust flagged record is the laundering signature. # Surface it loudly; silent demotion would hide the attack. raise ProvenanceDriftError( f"record {record_id}: flagged (s={write_anomaly_score:.2f}) " f"but provenance {provenance_weight:.2f} exceeds median " f"{PROVENANCE_MEDIAN:.2f}; possible trust laundering" ) return PROVENANCE_CAP # Not flagged: legitimate-trust regime. Do not cap. return provenance_weight def rank_recall( candidates: list[tuple[str, float, float]], ) -> list[tuple[str, float]]: """Rank (record_id, anomaly, provenance) by effective recall weight.""" ranked = [] for record_id, anomaly, provenance in candidates: try: weight = effective_recall_weight(anomaly, provenance, record_id) except ProvenanceDriftError: # Quarantine: exclude from ranking, surface for audit. weight = 0.0 ranked.append((record_id, weight)) return sorted(ranked, key=lambda kv: kv[1], reverse=True) ```