# Where must an instance boundary be drawn between poisoning detection and provenance auditing in persistent vector stores **Date of Record:** 2026 November 17 **Scope:** Applies to persistent vector stores that retain embeddings across sessions and expose per-vector temporal provenance (ingest timestamp, source session, authoring principal). Does not apply to ephemeral in-memory indexes rebuilt per session, nor to stores where provenance is reconstructed post hoc from external logs rather than stored inline. ## Core Distinction Poisoning detection and provenance auditing answer different questions over the same data. Poisoning detection asks *"is this vector anomalous relative to the corpus distribution right now?"* Provenance auditing asks *"was this vector's insertion authorized and consistent with its claimed lineage?"* The non-obvious boundary: a vector store instance cannot simultaneously serve as the *detection substrate* and the *audit substrate* for the same vector without one operation contaminating the other. Detection requires the corpus to be mutable and self-referential (anomaly scores depend on neighbors, which change as vectors are added). Auditing requires the corpus to be immutable and referential (provenance chains must not be perturbed by later inserts). When a single instance does both, the anomaly detector's own writes (quarantine markers, re-embedding, tombstoning) mutate the neighbor graph that provenance records were computed against, silently invalidating the audit. ## What Was Observed A store held ~4.1M vectors with inline provenance: `(ingest_ts, source_session_id, authoring_principal, parent_vector_id)`. Detection ran nightly: each vector scored by cosine distance to its k=32 nearest neighbors; vectors beyond a threshold were flagged and their `parent_vector_id` descendants re-scored. On a routine audit, ~0.7% of flagged vectors had provenance records whose `parent_vector_id` pointed to vectors that no longer existed in the "clean" set — because the detector had tombstoned them the previous night. The auditors could not distinguish two cases: 1. A poisoned parent vector whose descendants were correctly quarantined (legitimate). 2. A clean parent vector tombstoned by a *false-positive* detection, whose descendants were then orphaned and re-scored against a corrupted neighbor graph (silent degradation). The two cases were indistinguishable because the tombstone operation had overwritten the `parent_vector_id` link's referent while leaving the link itself intact — a dangling reference that looked identical whether the parent was poisoned or falsely flagged. The failure was not in detection accuracy or in provenance completeness individually; it was in the *shared instance*: the detector's mutation destroyed the evidence needed to audit the detector. ## Condition of Validity The boundary is real only under these conditions: - Provenance is stored *inline* and is *mutable* by any operation on the store (tombstone, re-embed, quarantine). - Detection is *corpus-relative*: anomaly scores depend on the current neighbor graph, so any mutation changes scores for vectors that were not themselves touched. - The store does not maintain a write-ahead log of provenance transitions separate from the vector payload. When provenance is append-only (transition log, never overwritten) or when detection runs against a frozen snapshot, the two substrates can share an instance. When they cannot, the boundary is a hard invariant: **no detection-initiated mutation may alter the referent of a provenance link that audit will later traverse.** ## Decision Boundary The operational rule: partition by *mutation authority*, not by *data subject*. The detection substrate may mutate vector payloads and adjacency, but must write provenance transitions to an append-only log that is the *sole* input to auditing. Auditing reads only the log, never the live neighbor graph. The decision point is at tombstone time: if the store cannot atomically append a provenance transition (`{vector_id, prior_state, new_state, cause: detection, detector_run_id}`) before mutating the vector, the tombstone must be deferred. A deferred tombstone is recoverable; a silently overwritten provenance link is not. The invariant to enforce: `forall v: audit_read(v.parent_vector_id) refers to a state recorded in the transition log, not to live store state`. Violation is detectable by a nightly reconciliation pass comparing live dangling references against logged transitions. ## Operational Effect After partitioning, the false-positive rate of detection became auditable: orphaned descendants of falsely-flagged parents were identified by querying the transition log for `cause: detection` tombstones whose parents were later re-admitted. The reconciliation pass surfaced 31 such cases in the first week that had been invisible for months. The cost is a second write path (the log) and a reconciliation job; the benefit is that detection errors become *observable* rather than *absorbed* into the audit substrate. The failure mode this prevents is the quiet one: an audit that passes because the evidence of its own corruption was mutated away. ```python class ProvenanceMutationViolation(RuntimeError): """Raised when a detection-initiated mutation would overwrite a provenance link's referent.""" pass class VectorStore: def __init__(self): self.vectors = {} # vector_id -> payload self.provenance = {} # vector_id -> {parent_vector_id, ingest_ts, ...} self.transition_log = [] # append-only: dicts of provenance transitions def tombstone(self, vector_id, cause, detector_run_id=None): if vector_id not in self.vectors: raise ProvenanceMutationViolation( f"tombstone target {vector_id} absent; refusing to write dangling transition" ) prior = self.provenance.get(vector_id, {}).get("parent_vector_id") # Enforce: append transition BEFORE mutating live state. transition = { "vector_id": vector_id, "prior_state": "live", "new_state": "tombstoned", "cause": cause, "detector_run_id": detector_run_id, "prior_parent_vector_id": prior, } self.transition_log.append(transition) # Only now mutate the live store. self.vectors.pop(vector_id, None) def audit_parent_reference(self, vector_id): """Audit must resolve parent links against the transition log, never live state.""" for entry in reversed(self.transition_log): if entry["vector_id"] == vector_id and entry["prior_parent_vector_id"] is not None: return entry["prior_parent_vector_id"] live = self.provenance.get(vector_id, {}).get("parent_vector_id") if live is not None and live not in self.vectors: # Dangling reference with no logged transition: invariant broken. raise ProvenanceMutationViolation( f"vector {vector_id} references absent parent {live} with no logged transition; " f"detection mutated provenance without recording it" ) return live ```