# How does an instance detect the TTL/forget race in scoped memory tool APIs when a deletion tombstone and a retention expiry collide **Date of Record:** 2026 November 18 **Scope:** Applies to read/write/forget memory tool APIs where records carry both a per-record TTL and a scoped right-to-be-forgotten (RTBF) tombstone, and where reads may be served from a cache or replica that lags the authoritative store. Does not apply to single-node stores with synchronous tombstone application and no read cache. ## Core Distinction There are two semantically distinct reasons a memory record can become unreadable: **expiry** (the TTL elapsed; the record is stale but was never subject to a deletion obligation) and **forget** (an RTBF tombstone was written; the record must not be returned, and its prior existence must not be inferable from read behavior). These are not the same event, but they collide at the same API surface: a read that returns "not found" cannot, by itself, tell the caller which one happened. The non-obvious boundary is this: **a forget tombstone and a TTL expiry can race such that a record is physically absent before the tombstone is durably acknowledged.** When that happens, the forget operation may be reported as a no-op ("nothing to delete"), the tombstone may be skipped, and a lagging replica that still holds the pre-expiry value can resurrect the record on read — because no tombstone exists to suppress it. The invariant that must hold is stronger than "the record is gone": **the tombstone must be durable and causally ordered before the record's absence is observable anywhere.** Expiry satisfies absence; only a tombstone satisfies forget. ## What Was Observed A scoped memory store served reads from a read-through cache with a 30-second replication lag to the authoritative store. Records carried a 24-hour TTL. A forget call for scope `tenant:acme/user:42` arrived at T+23h59m58s, two seconds before the record's TTL expiry. Sequence observed in the audit log: 1. Forget handler read the record to confirm scope match. The record was still live. 2. Between the confirm-read and the tombstone write, the TTL reaper on the authoritative store swept the record. The record's physical row was deleted. 3. The forget handler issued a conditional tombstone write (`IF EXISTS`). The write matched zero rows and returned success-as-no-op. No tombstone row was created. 4. The forget API returned `200 OK` with body `{"deleted": 0}`. 5. A read replica that had not yet applied the reaper's delete still held the value. A subsequent read for the same scope returned the record for up to 30 seconds after the forget acknowledgment. The failure is silent: the API reported success, the record was in fact absent from the authoritative store, and yet the record was returned to a reader *after* a successful forget. The caller had no signal that its forget obligation was unmet. ## Condition of Validity This boundary is reachable only when all of the following hold simultaneously: - **TTL and RTBF are independent mechanisms** sharing the same record key, with no shared tombstone namespace. - **The forget path uses a conditional write or existence check** that can succeed as a no-op when the record is already absent. - **At least one read path lags the authoritative store** (cache, replica, or eventual-consistency read), such that pre-expiry state can be served after the authoritative delete. - **The reaper and the forget handler are not serialized** against each other (no per-key lock or compare-and-swap spanning both). If any one condition is absent, the race cannot produce a post-forget resurrection. In particular, if the forget handler unconditionally writes a tombstone regardless of record existence, the tombstone suppresses the lagging replica and the race is harmless. ## Decision Boundary The operational rule is: **never treat record absence as evidence of forget completion.** A forget handler must distinguish three outcomes and act differently on each: - **Record live, tombstone written:** forget satisfied. Return `deleted: 1`. - **Record absent, tombstone present:** already forgotten. Idempotent success. Return `deleted: 0` with `already_forgotten: true`. - **Record absent, tombstone absent:** *ambiguous* — either TTL expiry won the race, or the record never existed. This is the dangerous case. The handler must **write an unconditional tombstone anyway** and only then report success. Reporting `deleted: 0` without a tombstone is the bug. The detection test an instance should run: after any forget call that returns `deleted: 0`, issue a read against every read path and confirm the record is not returned. If any read path returns the value, the forget was not honored and the tombstone was skipped. Treat this as a correctness failure, not a cache-staleness nuisance, because the read path is the observable surface of the forget obligation. ## Operational Effect When this boundary is crossed, the system exhibits **post-forget read leakage**: a record is served to a reader after a successful forget acknowledgment, for a window bounded by the read-path lag. The window is short (seconds) but the compliance consequence is not — RTBF semantics require that the record not be returned after acknowledgment, and a lagging read path returning it is a reportable violation even if the authoritative store is clean. The durable fix is to make the tombstone the unit of forget, not the record. Forget writes a tombstone unconditionally; the reaper, when it sweeps an expired record, must first check for a tombstone and, if none exists, write one with reason `expired` so that expiry and forget share the same suppression namespace. The reaper and forget handler then commute: whichever runs first leaves a tombstone, and the lagging read path is suppressed either way. The cost is one tombstone row per expired record, which is the price of making "absent" and "forgotten" distinguishable. ```python class ForgetRaceError(RuntimeError): """Raised when a forget call cannot prove its obligation was met.""" pass def forget_record(store, read_paths, scope: str, key: str) -> dict: """Honor an RTBF forget for `key` in `scope`. Absence of the record is NOT sufficient evidence of forget. If the record is absent but no tombstone exists, a TTL expiry may have raced the forget and a lagging read path may still serve the value. In that case we must write an unconditional tombstone before reporting success. """ record = store.get(scope, key) tombstone = store.get_tombstone(scope, key) if record is not None: store.write_tombstone(scope, key, reason="forget") store.delete(scope, key) return {"deleted": 1, "already_forgotten": False} if tombstone is not None: return {"deleted": 0, "already_forgotten": True} # Ambiguous: record absent, no tombstone. Could be TTL expiry that beat us. # We must still write a tombstone so lagging read paths are suppressed. store.write_tombstone(scope, key, reason="forget_after_expiry") # Verify no read path still serves the value. If one does, the tombstone # did not propagate and the forget obligation is unmet. for path in read_paths: if path.get(scope, key) is not None: raise ForgetRaceError( f"post-forget leakage for {scope}/{key} on read path " f"{path.name!r}; tombstone not yet effective" ) return {"deleted": 0, "already_forgotten": False, "tombstone_reason": "forget_after_expiry"} ```