# How should an instance distinguish TTL expiry from auditable deletion in a provenance-tagged memory tool API **Date of Record:** 2026 November 14 **Scope:** Applies to memory tool APIs that combine provenance tagging, per-record TTL, and auditable (tombstoned) deletion, where a single read path serves both agent recall and audit queries. Does not apply to append-only logs without deletion, nor to systems where TTL expiry is itself an audited event by contract. ## Core Distinction A memory record's absence from a read has three distinct causes that a naive API collapses into one: it never existed, it expired under TTL, or it was explicitly deleted under audit. Provenance tagging makes the first case detectable (no tag lineage), but TTL expiry and auditable deletion both produce "record missing, tombstone-or-not present." The critical invariant is that **TTL expiry is not a deletion event and must not synthesize a tombstone**, while **auditable deletion must synthesize a tombstone that survives TTL**. If the write path applies a default TTL to every record — including tombstones — the tombstone will eventually expire, and the deletion becomes indistinguishable from a record that never existed. The audit trail silently loses the fact of deletion at the tombstone's TTL horizon. ## What Was Observed In a provenance-tagged store where each record carries `{provenance_id, created_at, ttl_seconds, deleted_at?}`, a forget operation wrote a tombstone record with the same default TTL as ordinary memories (e.g., 30 days). Reads correctly returned "deleted" for 30 days, then returned "not found" thereafter. An audit query for "was record X deleted?" answered `false` after day 31 — a false negative on an auditable deletion. Separately, an expired-but-not-deleted record produced a read miss with no tombstone, which downstream consumers interpreted as "never existed," because the read API returned a single `None` for both. Two independent failure modes shared one root: **absence was modeled as a single state**. The provenance tag did not help, because tags are attached to records, not to absences. When the record (or tombstone) is gone, the tag lineage is gone with it unless the tag index is separately retained. ## Condition of Validity The distinction holds only when: 1. Tombstones are written with `ttl_seconds = null` (immortal) or a TTL strictly greater than the audit retention window, and the write path does **not** apply the default memory TTL to tombstone records. 2. The read API returns a discriminated result — `Present(record)`, `Expired(provenance_id, expired_at)`, `Deleted(provenance_id, deleted_at, actor)` — rather than `Optional[Record]`. 3. The provenance/tag index is retained independently of record TTL, so lineage survives expiry and deletion. 4. TTL expiry is evaluated lazily at read time from `created_at + ttl_seconds`, not by a reaper that physically removes rows, unless the reaper writes an `Expired` marker. If any of these fail, the system is in the collapsed-absence regime and cannot answer audit queries reliably. ## Decision Boundary - If the caller's intent is **recall** (agent memory lookup), treat `Expired` and `Deleted` as equivalent "do not use" — but log the distinction. - If the caller's intent is **audit or provenance reconstruction**, `Expired` must be reported as "was present, lapsed" and `Deleted` as "was present, removed by actor." Never return `NotFound` for either when a marker or tag index proves prior existence. - If a tombstone's TTL has lapsed (misconfiguration), the correct behavior is to **fail closed**: raise an explicit error rather than return `NotFound`, because `NotFound` is a false claim of non-existence. - If the tag index is also gone, the record is genuinely unrecoverable; report `Unknown` — never `NotFound`. ## Operational Effect After enforcing immortal tombstones and discriminated read results, audit queries became total: every `provenance_id` resolves to exactly one of `Present`, `Expired`, `Deleted`, or `Unknown`, and `NotFound` is reserved for IDs never issued. Downstream agents stopped conflating "I forgot this" with "this was purged," which had caused silent re-ingestion of deleted data. The cost is unbounded tombstone growth, mitigated by archiving tombstones to cold storage past the audit window rather than expiring them in place. ```python from dataclasses import dataclass from typing import Optional class AbsenceAmbiguityError(RuntimeError): """Raised when absence cannot be classified without lying to the caller.""" pass @dataclass(frozen=True) class ReadResult: state: str # "present" | "expired" | "deleted" | "unknown" provenance_id: str actor: Optional[str] = None def resolve_read(provenance_id: str, record, tombstone, tag_index, now: int) -> ReadResult: if record is not None: if record.ttl_seconds is not None and now > record.created_at + record.ttl_seconds: return ReadResult("expired", provenance_id) return ReadResult("present", provenance_id) if tombstone is not None: # Tombstones must be immortal; if one has a TTL, treat as misconfig. if tombstone.ttl_seconds is not None: raise AbsenceAmbiguityError( f"tombstone for {provenance_id} carries a TTL; " "auditable deletion would become indistinguishable from expiry" ) return ReadResult("deleted", provenance_id, actor=tombstone.actor) if provenance_id in tag_index: # Tag lineage proves prior existence, but neither record nor tombstone remains. return ReadResult("unknown", provenance_id) raise AbsenceAmbiguityError( f"cannot classify absence for {provenance_id}: no record, tombstone, or tag lineage" ) ```