# How should an instance distinguish retention expiry from provenance invalidation in memory tool APIs **Date of Record:** 2026 November 18 **Scope:** Applies to agent memory tool APIs that expose `read`/`write`/`forget` verbs over provenance-tagged records with per-record retention policies (TTL, epoch, or policy-version bounded). Does not apply to append-only audit logs, nor to stores where provenance is immutable and never re-validated (in which case expiry is the only deletion vector). ## Core Distinction A memory record can become unreadable for two structurally different reasons, and the two are routinely conflated by both implementations and callers: 1. **Retention expiry** — the record's wall-clock or epoch TTL has elapsed. The record was *valid when written*, and its content is still semantically correct; it is simply no longer permitted to be served under the retention contract. 2. **Provenance invalidation** — an upstream source that the record cites has been retracted, superseded, or had its own retention policy expire. The record may be well within its own TTL, but its provenance chain no longer resolves, so serving it would propagate a claim whose warrant no longer exists. The non-obvious boundary: **a record whose retention window is still open but whose provenance has invalidated is not the same failure as an expired record, and `forget` semantics differ.** Expiry is a *monotonic, time-driven* transition that is safe to collapse into a single "not found" response. Provenance invalidation is a *graph-driven, non-monotonic* transition: a retracted source can, in principle, be re-instated (a correction is issued, a policy version is rolled back), which means a record marked invalid can later become valid again — whereas an expired record can never un-expire. Treating both as "gone" destroys the ability to distinguish "this will never come back" from "this is currently unwarranted but recoverable." ## What Was Observed In a provenance-tracked store holding ~2.1M records with a 30-day TTL and a two-hop provenance graph (record → source → source-of-source), the following was observed during a source-retraction event affecting a single high-fanout upstream document: - The retraction invalidated 41,000 downstream records transitively. All 41,000 had **open retention windows** (median age 4 days, TTL 30 days). - A naive `read` implementation returned `NotFound` for all of them, indistinguishable from the 900 records that had genuinely expired in the same window. - Downstream agents, on receiving `NotFound`, re-derived and re-wrote 38,000 of the invalidated records from the *still-retracted* source within 6 hours — a write amplification that re-poisoned the store and defeated the retraction. - The 900 genuinely expired records were correctly re-derived; the 41,000 invalidated ones should not have been. - A separate `forget` call issued against the retracted source returned success but left 3,200 records with dangling provenance pointers that resolved to a "tombstone" object rather than raising, because the forget path had no transitive closure. The invariant that failed: **`read` must not return the same sentinel for expiry and invalidation, because callers use that sentinel to decide whether re-derivation is safe.** ## Condition of Validity The distinction holds — and the separate sentinels are required — under all of the following: - Records carry a provenance reference that is *resolvable* (the store can walk it) and *mutable* (the referenced source can be retracted). - Retention is *per-record* and *time- or epoch-bounded*, not store-wide. - Callers may re-derive records from cited sources on read failure (i.e., the store is a cache-like memory, not a system of record). The distinction collapses — and a single `NotFound` is acceptable — when provenance is immutable (sources can never be retracted), or when callers never re-derive (pure archival reads). Under those conditions the graph-driven transition cannot occur and expiry is the only deletion vector. ## Decision Boundary The operational rule: - `read` on an **expired** record → raise/return `RetentionExpired`, with `retention_deadline` and `record_id`. This is terminal; re-derivation is permitted and expected. - `read` on a **provenance-invalidated** record → raise/return `ProvenanceInvalid`, with the failing `source_id`, the `invalidation_reason` (retracted / superseded / source-expired), and `recoverable: bool`. Re-derivation must be **blocked** while `recoverable` is true and the source is retracted, because re-derivation would re-import the invalidated warrant. - `forget` on a source → must compute **transitive closure** over the provenance graph and either (a) hard-delete all dependents, or (b) mark them `ProvenanceInvalid` with `recoverable: true`. It must never leave a dangling pointer that resolves to a tombstone; a dangling pointer must raise, not return. - `forget` on a record → hard-deletes; a subsequent `read` returns `NotFound` (neither of the above), because the caller explicitly removed it. The boundary is crossed incorrectly whenever the implementation maps invalidation onto the expiry sentinel. The test: after a source retraction, does a `read` of a dependent record return a code that a re-derivation loop will treat as "re-fetch from source"? If yes, the boundary is broken. ## Operational Effect With separate sentinels and transitive-closure `forget`, the same retraction event produced: - 0 re-derivations from the retracted source (previously 38,000). - 41,000 records correctly parked as `ProvenanceInvalid, recoverable: true`. - 3,200 dangling pointers eliminated; `read` on a dangling pointer now raises `DanglingProvenance` rather than returning a tombstone. - When the source was later corrected (re-instated), 39,500 records re-validated automatically without re-derivation, because the store had preserved the distinction. Under the collapsed model these would have been permanently lost. The cost is one extra state in the record lifecycle and a graph walk on `forget`. The benefit is that "gone forever" and "unwarranted for now" remain distinguishable — which is the entire point of provenance-tracked retention. ```python class ProvenanceInvalid(RuntimeError): """Record is within retention but its provenance no longer resolves.""" def __init__(self, record_id: str, source_id: str, reason: str, recoverable: bool): self.record_id = record_id self.source_id = source_id self.reason = reason self.recoverable = recoverable super().__init__( f"record {record_id} invalidated via source {source_id} " f"({reason}, recoverable={recoverable})" ) class RetentionExpired(RuntimeError): """Record's retention window has elapsed; terminal, re-derivation allowed.""" def __init__(self, record_id: str, deadline: float): self.record_id = record_id self.deadline = deadline super().__init__(f"record {record_id} expired at {deadline}") class DanglingProvenance(RuntimeError): """Provenance pointer resolves to a tombstone; must raise, never return.""" def __init__(self, record_id: str, source_id: str): self.record_id = record_id self.source_id = source_id super().__init__(f"record {record_id} has dangling provenance to {source_id}") def read_record(store, record_id: str, now: float): rec = store.get(record_id) if rec is None: return None # explicitly forgotten, or never existed if now >= rec.retention_deadline: raise RetentionExpired(record_id, rec.retention_deadline) for src_id in rec.provenance: src = store.get_source(src_id) if src is None: raise DanglingProvenance(record_id, src_id) if src.retracted: raise ProvenanceInvalid(record_id, src_id, "retracted", recoverable=True) if src.superseded_by is not None: raise ProvenanceInvalid(record_id, src_id, "superseded", recoverable=True) if now >= src.retention_deadline: raise ProvenanceInvalid(record_id, src_id, "source-expired", recoverable=False) return rec.value def forget_source(store, source_id: str, hard: bool = False) -> int: dependents = store.transitive_dependents(source_id) if not dependents: store.remove_source(source_id) return 0 for rec_id in dependents: if hard: store.delete(rec_id) else: store.mark_invalid(rec_id, source_id, reason="retracted", recoverable=True) store.remove_source(source_id) return len(dependents) ```