# Which invariants prevent TTL expiry from being treated as deletion in scoped memory APIs **Date of Record:** 2026 November 17 **Scope:** Applies to memory tool APIs that expose `read`, `write`, and `forget` operations over scoped namespaces with per-entry TTL and an auditable deletion ledger. Does not apply to APIs where expiry and deletion share a single tombstone record by design, nor to unscoped key-value stores with no audit contract. ## Core Distinction The failure mode is conflating two lifecycle events that look identical to a reader but are semantically disjoint to an auditor: - **Expiry**: a TTL clock crossed a threshold. The entry becomes invisible to `read`. No actor requested removal. - **Deletion**: a caller invoked `forget` (or an authorized retention policy did so on its behalf). A principal is accountable, and the audit ledger must record that principal. A memory API can satisfy read semantics perfectly while silently violating its audit contract if expiry writes the same tombstone shape as deletion. The invariant that prevents this is: *every entry removed from the readable set must emit a ledger record whose `cause` field is drawn from a closed enumeration, and `cause=ttl_expiry` must never carry a `principal`*. Expiry is anonymous; deletion is attributed. Collapsing them makes the ledger unable to answer "who deleted this," which is the only question the ledger exists to answer. ## What Was Observed A scoped memory service used a single `tombstone(namespace, key, ts)` record for both paths. A `read` on an expired key triggered lazy reaping, which wrote a tombstone. A subsequent `forget` on the same key found the tombstone present, short-circuited as idempotent, and wrote nothing. The audit log therefore contained a tombstone with no principal for a key that a human operator had explicitly requested be forgotten. During a data-subject deletion audit, the operator could not prove the deletion had occurred on request; the only evidence was an anonymous expiry tombstone timestamped before the forget call. The forget was real, the data was gone, and the audit was wrong. A second, subtler instance: a namespace-wide retention sweep expired 40k entries and wrote 40k tombstones. The ledger's `principal` column was null for all of them. A downstream compliance job that counted "deletions by principal" reported zero deletions for the period, masking a genuine operator-initiated forget of 12 keys that had raced the sweep and been absorbed into the anonymous batch. ## Condition of Validity The invariant holds only under these conditions, and each is independently checkable: 1. **Causal separation at write time.** Expiry and forget must produce distinguishable ledger records. A `cause` enum (`ttl_expiry`, `explicit_forget`, `policy_retention`) is the minimum. A nullable `principal` is acceptable only if the schema enforces `principal IS NULL` iff `cause = ttl_expiry`. 2. **No short-circuit absorption.** `forget` on an already-expired-but-not-yet-reaped key must still emit an `explicit_forget` record. Idempotency of the *effect* (data is gone) must not suppress the *attestation* (a principal asked). These are different guarantees. 3. **Reap ordering is not authoritative.** Lazy reaping on `read` must not be the sole writer of tombstones for keys that a `forget` will later target. Either reaping is eager and ledger-first, or `forget` writes its attestation regardless of reap state. 4. **Clock provenance.** TTL expiry timestamps come from the service clock; forget timestamps come from the request. Mixing them in one column without a `clock_source` field makes ordering assertions unverifiable across clock skew. If any condition fails, the ledger is a log of *absence*, not a log of *deletion*, and cannot back an auditable deletion guarantee. ## Decision Boundary Choose the split-ledger design when: the API carries a compliance or data-subject-deletion obligation, `forget` is callable by more than one principal, or TTLs are short enough that expiry and forget routinely race. Choose the unified tombstone only when the audit contract is explicitly "absence is sufficient" and no principal attribution is promised—and then document that `forget` is advisory, not attested. The boundary is sharp: the moment any consumer asks "who removed this and why," unified tombstones are disqualified. Do not attempt to reconstruct attribution post hoc by correlating request logs with tombstone timestamps; the race in the second observation makes that reconstruction provably ambiguous. ## Operational Effect Implementing the split ledger changes `forget` from a conditional delete into an unconditional attestation write followed by a best-effort physical removal. Cost: one extra ledger row per forget even when the key is already gone. Benefit: the ledger answers attribution questions under every interleaving, and retention sweeps become separable from operator actions in aggregate reporting. The 40k-sweep masking incident becomes impossible because sweep records carry `cause=policy_retention` and forget records carry `cause=explicit_forget` with a principal, and no code path can emit one as the other. ```python from dataclasses import dataclass from enum import Enum from typing import Optional class Cause(str, Enum): TTL_EXPIRY = "ttl_expiry" EXPLICIT_FORGET = "explicit_forget" POLICY_RETENTION = "policy_retention" class LedgerIntegrityError(RuntimeError): """Raised when a ledger record violates the expiry/deletion attribution invariant.""" @dataclass(frozen=True) class LedgerRecord: namespace: str key: str cause: Cause principal: Optional[str] clock_source: str # "service" for expiry, "request" for forget def _validate(rec: LedgerRecord) -> None: # Invariant: expiry is anonymous, deletion is attributed. if rec.cause is Cause.TTL_EXPIRY: if rec.principal is not None: raise LedgerIntegrityError( f"ttl_expiry for {rec.namespace}/{rec.key} carried principal " f"{rec.principal!r}; expiry must be anonymous" ) if rec.clock_source != "service": raise LedgerIntegrityError( f"ttl_expiry for {rec.namespace}/{rec.key} used clock_source " f"{rec.clock_source!r}; expiry must use the service clock" ) else: if rec.principal is None: raise LedgerIntegrityError( f"{rec.cause.value} for {rec.namespace}/{rec.key} has no principal; " f"attributed deletion requires an accountable actor" ) if rec.clock_source != "request": raise LedgerIntegrityError( f"{rec.cause.value} for {rec.namespace}/{rec.key} used clock_source " f"{rec.clock_source!r}; attributed deletion must use the request clock" ) def forget(namespace: str, key: str, principal: str, ledger: list[LedgerRecord]) -> None: """Attest an explicit forget. Writes the record unconditionally, even if the key was already reaped by TTL, so idempotency of effect never suppresses attestation.""" if not principal: raise LedgerIntegrityError("forget requires a non-empty principal") rec = LedgerRecord( namespace=namespace, key=key, cause=Cause.EXPLICIT_FORGET, principal=principal, clock_source="request", ) _validate(rec) ledger.append(rec) # Physical removal is best-effort and independent of the attestation above. def reap_expired(namespace: str, key: str, ledger: list[LedgerRecord]) -> None: rec = LedgerRecord( namespace=namespace, key=key, cause=Cause.TTL_EXPIRY, principal=None, clock_source="service", ) _validate(rec) ledger.append(rec) def assert_attributed(ledger: list[LedgerRecord], namespace: str, key: str) -> None: """Fail closed if a forget was requested but only anonymous expiry is on record.""" for rec in ledger: if rec.namespace == namespace and rec.key == key: if rec.cause is Cause.EXPLICIT_FORGET and rec.principal: return raise LedgerIntegrityError( f"no attributed deletion on record for {namespace}/{key}; " f"expiry tombstones cannot satisfy a forget attestation" ) ```