# Which invariants prevent scoped-forget APIs from silently resurrecting deleted memory through derived indexes **Date of Record:** 2026 November 15 **Scope:** Applies to read/write/forget memory tool APIs where deletion is scoped (by principal, tag, or time window) and where reads are served through one or more derived structures (vector indexes, summary caches, inverted indices, embedding clusters) rather than directly from the authoritative store. Does not apply to single-store systems with no derived index, nor to whole-namespace wipe operations where the scope covers every derived structure by construction. ## Core Distinction A scoped `forget(scope)` call is a request to make a *set* of records unreadable through *every* read path. The authoritative record store and the derived indexes are two different read paths. The invariant that matters is not "the record is deleted" but "no read path can reconstruct the record after deletion." These diverge precisely when the derived index is updated asynchronously, when it stores lossy but invertible derivatives (chunk embeddings, token n-grams, summarised fragments), or when the index is shared across scopes and only filtered at query time. In all three cases the authoritative delete succeeds and the read path still answers. The failure is non-obvious because the API contract is stated in terms of records, while the leak occurs in terms of *derivatives of records*. A deleted record's embedding vector is not the record, but a nearest-neighbour query over it returns the record's chunk text from the index's payload, or returns a neighbour whose payload contains the deleted span. Deletion of the row does not delete the vector; deletion of the vector does not delete the payload; deletion of the payload does not delete the summary cache that was built from it. ## What Was Observed A memory service exposed `write(principal, text)`, `read(principal, query)`, and `forget(principal, predicate)`. Forgets were applied to the primary table and returned `{"deleted": N}` synchronously. Reads were served by a hybrid retriever: a vector index over chunk embeddings plus a keyword index over token n-grams, with a post-retrieval filter that dropped chunks whose `principal` no longer matched a live row. After a forget, direct reads returned nothing — the post-filter correctly dropped orphaned chunks. But two leaks persisted: 1. **Aggregate reconstruction.** The keyword index retained per-term document frequencies. A query for a rare term returned zero chunks after filtering, but the response metadata included `total_matches` computed pre-filter. Repeated probing of rare terms reconstructed the existence and rough count of forgotten documents without ever returning their text. The post-filter changed *what was returned*, not *what was counted*. 2. **Neighbour contamination.** The vector index retained embeddings. A query embedding near a forgotten chunk returned live neighbours whose payload included a shared summary fragment that had been generated *from* the forgotten chunk during ingestion. The forgotten text was recoverable through a live record's derived payload. A third, subtler leak appeared during reindex: the summary cache was keyed by content hash, so re-ingesting a near-duplicate of forgotten content hit the cache and restored the summary without ever re-reading the authoritative store. The cache had no scope dimension and no tombstone check. ## Condition of Validity The invariant "forget is complete" holds only if, for every derived structure `D` and every read path `R` that consults `D`, one of the following is true: - `D` is updated in the same transaction/epoch as the authoritative delete, **and** `R` filters on the authoritative tombstone *before* computing any aggregate or neighbour expansion, not after; or - `D` carries a per-record tombstone that is consulted at the point of derivative use — including document-frequency counting, summary-cache lookup, and neighbour payload assembly; or - `D` is rebuilt from the authoritative store on a bounded schedule, and reads are served from a snapshot that is known to postdate the delete. The invariant fails when the filter is applied at output assembly rather than at derivative access. This is the general condition: **any derived quantity computed before the tombstone filter is a potential reconstruction channel.** Counting, ranking, neighbour expansion, cache lookup, and deduplication are all derived quantities. ## Decision Boundary The boundary between a sound and an unsound scoped-forget implementation is whether the tombstone is *consulted at derivative-use time* or *applied at result-emission time*. Concretely: - If `total_matches`, scores, or ranks are computed over the unfiltered index and the filter is applied to the returned list, the implementation is unsound for aggregates even if it is sound for text. - If neighbour expansion or summary assembly reads payloads from an index that has not seen the tombstone, the implementation is unsound for reconstructable spans even if direct hits are filtered. - If any cache is keyed on content rather than on `(scope, record_id)` and lacks a tombstone check, it will resurrect content on re-ingest. The decision to treat a system as compliant requires demonstrating that every derived quantity is computed post-tombstone, not merely that returned text is post-filtered. Absent that demonstration, treat the forget as advisory, not guaranteed. ## Operational Effect Operationally, this changes how forget must be tested and how guarantees must be stated. A forget test that asserts "direct read returns empty" passes on a system that leaks through aggregates and neighbours. The correct test asserts that *every* read path — including count metadata, ranking side channels, neighbour payloads, and cache hits on re-ingest — is invariant under the forget. In practice this forces either synchronous tombstone propagation into every derived structure, or a read-side tombstone check bound to derivative computation, or an epoch barrier that refuses reads until reindex catches up. The cost is that forget becomes a scope-wide barrier rather than a row delete, and reads that cross the barrier must either block or serve from a pre-delete snapshot with the tombstone applied at every derivative access point. Systems that treat forget as a cheap row operation and reads as a post-filter will pass naive tests and fail adversarial reconstruction. ```python class ScopedForgetViolation(RuntimeError): """Raised when a read path can reconstruct content outside the live scope.""" pass class MemoryRecord: def __init__(self, record_id, scope, text, tombstoned=False): self.record_id = record_id self.scope = scope self.text = text self.tombstoned = tombstoned class DerivedIndex: """Vector/keyword index that must consult tombstones at derivative-use time.""" def __init__(self, store): self.store = store self.vectors = {} # record_id -> embedding self.summary_cache = {} # content_hash -> summary fragment def _live(self, record_id): rec = self.store.get(record_id) return rec is not None and not rec.tombstoned def count_matches(self, term): # Aggregate must be computed post-tombstone, not pre-filtered. total = 0 for rid, payload in self.vectors.items(): if term in payload and self._live(rid): total += 1 return total def neighbours(self, query_vec, k): scored = [] for rid, vec in self.vectors.items(): if not self._live(rid): continue # tombstone consulted before neighbour expansion scored.append((self._similarity(query_vec, vec), rid)) scored.sort(reverse=True) out = [] for _, rid in scored[:k]: rec = self.store[rid] if not self._live(rid): raise ScopedForgetViolation( f"neighbour {rid} resurrected after forget" ) out.append(rec.text) return out def summary_for(self, content_hash, scope): # Cache is keyed on content and must be scope-checked on hit. cached = self.summary_cache.get(content_hash) if cached is not None: owner = cached["scope"] owner_rec = self.store.get(cached["record_id"]) if owner_rec is None or owner_rec.tombstoned or owner != scope: raise ScopedForgetViolation( f"summary cache hit for {content_hash} crosses scope {scope}" ) return cached["summary"] return None @staticmethod def _similarity(a, b): return sum(x * y for x, y in zip(a, b)) def forget(store, index, predicate): deleted = 0 for rid, rec in list(store.items()): if predicate(rec): rec.tombstoned = True deleted += 1 # Derived structures are not mutated here; reads must honour tombstones. return deleted def guarded_read(index, query_vec, term, scope): # A count computed over an untombstoned index is a reconstruction channel. if index.count_matches(term) > 0 and not any( index._live(rid) for rid in index.vectors ): raise ScopedForgetViolation( "aggregate count leaks forgotten records outside scope" ) return index.neighbours(query_vec, k=5) ```