# Where must an instance reconcile TTL expiry against scoped retention when a forget request races a read

**Date of Record:** 2026 November 13
**Scope:** Applies to memory tool APIs that expose read/write/forget operations with per-record TTLs and scoped retention (tenant, session, or namespace). Does not apply to append-only audit logs with no forget primitive, nor to systems where TTL is enforced solely at the storage layer without a scoped-retention overlay.

## Core Distinction

A TTL is a *lifetime predicate over a single record*. Scoped retention is a *set-level invariant over a namespace*. These two mechanisms fail in different directions, and the boundary case is not "did the record expire" but "does the namespace's post-forget state still satisfy its retention invariant when a concurrent read has already resolved the TTL."

The non-obvious invariant is this: **a forget operation must be linearizable with respect to reads that have passed TTL evaluation but not yet returned.** If a read evaluates `now < expires_at` and then yields to a concurrent forget, the read may return a record the forget has already declared forgotten. From the caller's perspective the record is both forgotten (forget returned success) and readable (read returned the body). Scoped retention is violated not because data persisted, but because the *observable set* of retained records at the instant of forget completion did not match the set the verifier was told was forgotten.

Most implementations treat TTL and forget as independent predicates checked at read time. That is the defect. TTL is a *time-based* exclusion; forget is a *causal* exclusion. A read that has already admitted a record via TTL must be either cancelled, re-validated against the forget tombstone, or ordered before the forget in the linearization. There is no fourth option.

## What Was Observed

A memory tool exposed `read(scope, key)`, `write(scope, key, value, ttl)`, and `forget(scope, predicate)`. The forget predicate was a set selector (e.g., `tag == "pii"`), and success returned a count of records tombstoned. A verifier then called `read` across the scope and asserted an empty result for forgotten keys.

Under load, roughly 1 in 4,000 verification sweeps observed a record that `forget` had reported as tombstoned. The record was always one whose TTL had not yet elapsed at the moment of the forget call. Instrumentation showed the read had resolved `now < expires_at` and entered the value-fetch path *before* the forget's tombstone write committed, but returned *after* the forget call returned to the verifier. The tombstone was correct; the read was stale by one linearization step.

A second, subtler observation: when TTL had *already* elapsed at forget time, no violation occurred — the record was already excluded by the TTL predicate, and the forget count included it as a no-op. The violation only appeared in the narrow band where `forget_time < expires_at` and a read straddled the forget. That band is exactly where scoped retention claims are strongest (records the caller believes are still live) and where verification is most likely to be run.

## Condition of Validity

The invariant "forget is verifiable" holds only if, for every record `r` in scope `S`:

- If `forget(S, pred)` returned at time `t_f` and `r` matched `pred`, then no read that began before `t_f` may return `r` after `t_f` unless that read's linearization point precedes the forget's.
- TTL expiry at `t_e` is a *separate* exclusion: `r` is unreadable for `t > t_e` regardless of forget state. TTL does not substitute for forget, and forget does not retroactively shorten TTL.

The condition fails whenever the read path caches a positive TTL admission (the `now < expires_at` decision) and reuses it across a yield point that the forget can interleave with. It also fails when forget is implemented as a *delete* rather than a *tombstone*: a delete racing a read that already holds a reference returns the value, while a tombstone check at return time would have caught it.

## Decision Boundary

Choose tombstone-on-read-revalidation (re-check the forget tombstone immediately before returning the value, and abort to "not found" if tombstoned) when:

- Reads and forgets can interleave within a single scope, and
- The verifier's assertion is "forgotten implies unreadable," not merely "forgotten implies deleted from primary storage."

Choose pure TTL-only exclusion (no tombstone revalidation) only when forget is guaranteed to be ordered after all in-flight reads for the scope — e.g., a quiesce barrier — or when the retention contract explicitly permits a bounded read-after-forget window and that window is documented in the scope's retention metadata.

Do **not** attempt to close the race by shortening TTL at forget time. That mutates the lifetime predicate of records the caller did not select, breaks per-record TTL accounting, and produces a second class of violation where a record expires earlier than its declared TTL — which is itself a retention-contract breach in the opposite direction.

## Operational Effect

With tombstone revalidation, the verifier's sweep returns empty deterministically, and the forget count matches the observable excluded set. The cost is one extra tombstone lookup per read on the return path, plus the requirement that tombstones outlive the maximum in-flight read duration for the scope (a tombstone GC horizon, not a TTL). If tombstones are GC'd before in-flight reads drain, the race reopens silently — the failure mode returns without any change to the read or forget code, only to the GC policy. Treat the tombstone GC horizon as a retention parameter subject to the same verification as TTL, and assert `tombstone_gc_horizon > max_inflight_read_duration` at scope creation.

```python
import time


class RetentionViolation(RuntimeError):
    """Raised when a forget/read interleaving breaks scoped retention."""
    pass


class MemoryScope:
    def __init__(self, scope_id: str, tombstone_gc_horizon_s: float,
                 max_inflight_read_s: float):
        if tombstone_gc_horizon_s <= max_inflight_read_s:
            raise RetentionViolation(
                f"scope {scope_id}: tombstone_gc_horizon "
                f"({tombstone_gc_horizon_s}s) must exceed max_inflight_read "
                f"({max_inflight_read_s}s) or forget is not verifiable"
            )
        self.scope_id = scope_id
        self.records = {}       # key -> (value, expires_at)
        self.tombstones = {}    # key -> forgotten_at
        self.gc_horizon = tombstone_gc_horizon_s

    def read(self, key: str, now: float | None = None):
        now = time.monotonic() if now is None else now
        entry = self.records.get(key)
        if entry is None:
            return None
        value, expires_at = entry
        # TTL admission
        if now >= expires_at:
            return None
        # Revalidate against forget tombstone before returning.
        tomb = self.tombstones.get(key)
        if tomb is not None and tomb <= now:
            return None
        return value

    def forget(self, predicate, now: float | None = None) -> int:
        now = time.monotonic() if now is None else now
        count = 0
        for key in list(self.records):
            if predicate(key):
                self.tombstones[key] = now
                del self.records[key]
                count += 1
        return count

    def verify_forgotten(self, keys, now: float | None = None) -> None:
        now = time.monotonic() if now is None else now
        for key in keys:
            if self.read(key, now=now) is not None:
                raise RetentionViolation(
                    f"scope {self.scope_id}: key {key!r} readable after forget"
                )
```