# What structural signals indicate an erasure-vs-TTL race in scoped memory tool APIs

**Date of Record:** 2026 November 12
**Scope:** Applies to read/write/forget memory tool APIs where records carry both a per-scope TTL and an independent right-to-be-forgotten (RTBF) erasure obligation. Does not apply to append-only audit logs exempt from RTBF, or to stores where TTL expiry and erasure share a single deletion path.

## Core Distinction

A TTL expiry and an RTBF erasure both remove a record, but they are not equivalent operations. TTL expiry is a *retention policy event*: the record's content becomes unreachable, but the store may legitimately retain tombstones, hashes, scope membership counters, or derived indexes for reconciliation. RTBF erasure is a *legal obligation event*: the obligation is that no reconstructable representation of the personal data survives in any scope, including derived indices, caches, and cross-scope replications.

The non-obvious boundary: **a record that expires by TTL shortly before an erasure request arrives can silently satisfy the erasure in the primary store while leaving derived artifacts (embedding caches, inverted-index postings, scope-summary counters) intact** — because the erasure path, seeing no live record, returns success without cascading to derived stores that were populated under the TTL regime and never registered as erasure targets.

## What Was Observed

In a memory tool with three scopes (`session`, `user`, `org`) and per-scope TTLs, the following sequence reproduced the failure:

1. `write(scope=user, key=k, ttl=3600)` — record replicated into an embedding cache and an inverted index keyed by `(scope, key)`.
2. TTL fires at t+3600. Primary record deleted; tombstone written; **derived indices not purged** (by design — they are considered eventually-consistent).
3. At t+3605, `forget(scope=user, key=k)` arrives as an RTBF request.
4. The forget handler probes the primary store, finds no live record, and returns `{"status": "already_absent"}`.
5. The embedding cache still contains a vector whose nearest-neighbor queries return the erased subject's attributes.

Structural signals that indicated the race *before* it manifested in a query:

- The forget response contained `already_absent` rather than `erased`. These are indistinguishable to a caller that only checks HTTP 200.
- The tombstone record's `reason` field was `ttl_expiry`, not `erasure`. Erasure handlers must treat a pre-existing tombstone as a *cascade trigger*, not as proof of completion.
- Derived-index watermarks lagged the primary store's deletion watermark by a measurable interval, and the forget path did not wait on that watermark.
- Scope counter deltas: the `user` scope's live-record count dropped at TTL time, but the index posting count did not drop correspondingly until a later compaction.

## Condition of Validity

The invariant `forget(key) ⟹ no reconstructable representation of key's content survives in any scope` holds **only if** the forget path treats `absent-in-primary` as a *precondition to cascade*, never as a *terminal success state*. It is violated whenever:

- TTL deletion and erasure share a "record gone" fast path.
- Derived stores (embeddings, inverted indices, cross-scope replicas, caches) are populated outside the erasure registry, so the erasure path cannot enumerate them.
- The API returns a single success status for both `erased` and `already_absent` without distinguishing whether cascade ran.

The invariant is *not* required when the store is append-only and exempt from RTBF, or when derived stores are provably rebuilt from the primary on every read (no persistence beyond TTL).

## Decision Boundary

When a `forget` request arrives and the primary record is absent, the correct decision is **not** "return success." The boundary is:

- If the tombstone's `reason == erasure` **and** the erasure cascade watermark is ≥ the tombstone's creation sequence, return `erased` (idempotent replay).
- If the tombstone's `reason == ttl_expiry`, or no tombstone exists, **run the cascade** against all derived stores registered for `(scope, key)`, then advance the erasure watermark, then return `erased`.
- If the derived-store set for `(scope, key)` cannot be enumerated, return a *failure*, not a success — the obligation is unsatisfiable, and the caller must be told.

The failure to avoid is treating absence as completion. Absence is a *reason to cascade*, not a *reason to stop*.

## Operational Effect

After the fix, the forget handler was changed to always issue a cascade regardless of primary-store presence, and to distinguish `erased` from `already_absent` in the response. The measurable effects:

- Forgets that previously returned in ~2 ms (fast path) now take ~15–40 ms (cascade across embedding cache and inverted index), which is acceptable for RTBF latency budgets.
- Idempotent replays remain fast because the erasure watermark short-circuits the cascade.
- A new failure mode surfaced: cascades against derived stores that were never populated for the key must not error — they must be treated as no-ops with a recorded `skipped` count, so the watermark still advances.

The transferable lesson: **any retention mechanism (TTL) that deletes ahead of an obligation mechanism (RTBF) creates a window where the obligation mechanism observes absence and mistakes it for fulfillment.** The structural signal is a success status that does not encode *which* deletion path ran. Agents operating memory tools should treat `already_absent` from a forget call as a *warning*, not a confirmation, unless the response explicitly states the cascade ran.

```python
class ErasureCascadeIncompleteError(RuntimeError):
    """Raised when a forget request cannot prove derived-store cascade ran."""
    pass


DERIVED_STORES = ("embedding_cache", "inverted_index", "scope_replica")


def forget(scope: str, key: str, store, watermark: dict) -> dict:
    tombstone = store.get_tombstone(scope, key)
    target_seq = store.current_sequence()

    # Replay of an already-completed erasure: safe to short-circuit.
    if tombstone and tombstone["reason"] == "erasure" and watermark.get((scope, key), -1) >= tombstone["seq"]:
        return {"status": "erased", "replayed": True}

    # Absent-in-primary is a cascade trigger, never terminal success.
    if tombstone is None and not store.has_live(scope, key):
        reason = "absent"
    elif tombstone and tombstone["reason"] == "ttl_expiry":
        reason = "ttl_expired"
    else:
        reason = "live"

    cascaded, skipped = [], []
    for derived in DERIVED_STORES:
        backend = store.derived(derived)
        if backend is None or not backend.registered(scope, key):
            raise ErasureCascadeIncompleteError(
                f"derived store {derived!r} cannot be enumerated for ({scope},{key}); "
                f"erasure obligation unsatisfiable, reason={reason}"
            )
        if backend.contains(scope, key):
            backend.purge(scope, key)
            cascaded.append(derived)
        else:
            skipped.append(derived)

    watermark[(scope, key)] = target_seq
    return {"status": "erased", "reason": reason, "cascaded": cascaded, "skipped": skipped}
```