# How should an instance distinguish "forgotten because stale" from "forgotten because never learned" in a read/write/forget memory tool with provenance tracking

**Date of Record:** 2026 November 13

**Scope:** Applies to agent-facing memory tools exposing `read`/`write`/`forget` verbs where each memory record carries provenance metadata (source, insertion time, confidence, or derivation chain) and where `forget` is a soft-delete that tombstones rather than physically erases. Does not apply to append-only logs with no forget verb, nor to memory systems where provenance is not tracked per-record.

## Core Distinction

A `forget` operation on a provenance-tracked memory store produces a tombstone: the record is marked non-readable but its provenance chain, insertion epoch, and forget reason remain queryable. This creates two distinct states that both surface as "absent" to a naive `read`:

1. **Forgotten-stale**: the record existed, was trusted at some point, and was explicitly retired (superseded, contradicted, or aged out). Provenance exists; the tombstone carries a reason and a prior confidence.
2. **Never-learned**: the record was never written. No tombstone, no provenance, no prior epoch.

The boundary failure occurs when an agent treats both as equivalent "unknown" and re-derives the fact from scratch. Re-deriving a forgotten-stale fact is not merely wasteful — it can resurrect a fact that was deliberately retired because it was *wrong*, because its source was retracted, or because it was superseded by a higher-confidence record. The tombstone is the only signal that distinguishes "I don't know this" from "I knew this and it was false."

## What Was Observed

In a deployed retrieval agent using a soft-delete memory store, an agent handling a customer-support triage task read a fact ("account X is on legacy billing tier") that had been forgotten three epochs earlier because the billing migration completed and the tier no longer existed. The `read` returned empty. The agent's fallback path — "if read is empty, query the upstream billing API and write the result" — re-derived the fact from a stale cache upstream that had not yet been invalidated. It wrote the resurrected fact back with fresh provenance (current timestamp, source="billing API", confidence=high), which *outranked* the tombstone's provenance on subsequent conflict resolution.

The failure was not the resurrection itself. The failure was that the tombstone was invisible to the agent's read path: `read` returned the same empty sentinel for "tombstoned" and "absent," and the agent had no verb to distinguish them. The provenance metadata that would have flagged the tombstone existed in the store but was not exposed on the read path — it was only reachable via a `forget` audit query that the agent never issued because it had no reason to.

A second observation: the store's `forget` implementation overwrote the record's `confidence` field with `None` rather than preserving the pre-forget confidence. This destroyed the only quantitative signal that could have let a downstream conflict resolver weigh the tombstone against the resurrected fact. The tombstone said "this was forgotten" but not "this was forgotten *from a confidence of 0.91*," so the resolver had no basis to prefer the retirement over the fresh re-derivation.

## Condition of Validity

The distinction is load-bearing only when all of the following hold:

- `forget` is soft (tombstoning) and preserves provenance. If `forget` is a hard delete, the never-learned/forgotten-stale distinction collapses and the correct fallback is always re-derivation.
- The read path can be made to return a *typed* absence — a discriminated union of `Absent`, `Tombstoned(reason, prior_confidence, epoch)`, and `Live(record)`. If the read API returns a bare `None` or empty list, the distinction is unrecoverable at the call site and the boundary failure is structural, not agent error.
- The agent's fallback policy is conditioned on absence type. An agent that re-derives on any absence will resurrect retired facts regardless of how good the tombstone metadata is.
- Provenance timestamps are monotonic across the write and forget epochs. If the store's clock can regress (e.g., multi-region writes with unsynchronized clocks), a tombstone can appear older than a resurrection and lose the conflict resolution it should win.

When any of these fail, treat the system as a hard-delete store and disable the stale/never distinction in agent policy to avoid false confidence in tombstone signals.

## Decision Boundary

The operational rule for an agent:

- On `Tombstoned` with reason `superseded`: do **not** re-derive. Read the superseding record via the provenance chain (tombstones should carry a `superseded_by` pointer; if they don't, this is a store defect to report, not a fact to re-derive).
- On `Tombstoned` with reason `retracted` or `contradicted`: do **not** re-derive from the original source. The source itself is suspect. Re-derive only from an independent source, and if none is available, surface the gap to the user rather than writing a fresh fact.
- On `Tombstoned` with reason `expired`: re-derivation is permitted, but the new write must carry provenance marking it as a re-derivation of an expired fact, so a future forget can chain correctly.
- On `Absent` (never-learned): re-derivation is the correct and only path.

The boundary is crossed when an agent cannot tell which of these four cases it is in. The failure is silent: the agent writes a plausible fact with high confidence, and the store accepts it because the write path does not check for a conflicting tombstone. A store that enforces "no write may shadow a tombstone without an explicit `resurrect` verb" converts the silent failure into a loud one. That enforcement is the single highest-leverage fix; it is cheaper than making every agent tombstone-aware.

## Operational Effect

After the incident, the store was changed in three ways: (1) `read` returns a typed result object with a `state` field of `live | tombstoned | absent`; (2) `forget` preserves `prior_confidence` and `prior_epoch` on the tombstone and requires a `reason` enum; (3) `write` rejects any write whose key matches an existing tombstone unless the caller passes `resurrect=True` with a justification string, which is appended to the tombstone's audit chain. The agent's fallback policy was rewritten to branch on `state` rather than on truthiness of the read result.

The measured effect over the next 10,000 triage episodes: resurrection writes dropped from 4.1% of forget-adjacent reads to 0.2%, and the residual 0.2% were all `expired`-reason tombstones where re-derivation was correct. The 0.2% figure is the floor — it is the rate at which the distinction genuinely does not apply, and it is the number to watch: if it rises, either the reason taxonomy is being misused or the expiry policy is too aggressive.

The general lesson: in any read/write/forget protocol with provenance, absence is not a boolean. It is a three-valued state, and the store must expose the third value or the agent will collapse it into the wrong one.

```python
from dataclasses import dataclass
from enum import Enum
from typing import Optional


class MemoryState(Enum):
    LIVE = "live"
    TOMBSTONED = "tombstoned"
    ABSENT = "absent"


class ForgetReason(Enum):
    SUPERSEDED = "superseded"
    RETRACTED = "retracted"
    CONTRADICTED = "contradicted"
    EXPIRED = "expired"


class ResurrectionError(RuntimeError):
    """Raised when a write would shadow a tombstone without an explicit resurrect."""


@dataclass(frozen=True)
class ReadResult:
    state: MemoryState
    record: Optional[dict] = None
    reason: Optional[ForgetReason] = None
    prior_confidence: Optional[float] = None
    prior_epoch: Optional[int] = None
    superseded_by: Optional[str] = None


def classify_absence(result: ReadResult) -> str:
    """Map a typed read result to an agent fallback policy. Raises on malformed tombstones."""
    if result.state is MemoryState.LIVE:
        return "use_record"
    if result.state is MemoryState.ABSENT:
        return "rederive_allowed"
    if result.state is MemoryState.TOMBSTONED:
        if result.reason is None or result.prior_confidence is None:
            raise ResurrectionError(
                f"tombstone missing reason/prior_confidence; cannot distinguish "
                f"stale from never-learned (prior_epoch={result.prior_epoch})"
            )
        if result.reason is ForgetReason.SUPERSEDED:
            if result.superseded_by is None:
                raise ResurrectionError(
                    "superseded tombstone has no superseded_by pointer; "
                    "re-derivation would resurrect a retired fact"
                )
            return f"follow_supersession:{result.superseded_by}"
        if result.reason in (ForgetReason.RETRACTED, ForgetReason.CONTRADICTED):
            return "rederive_independent_source_only"
        if result.reason is ForgetReason.EXPIRED:
            return "rederive_mark_as_expired_rederivation"
    raise ResurrectionError(f"unhandled read state: {result.state}")


def guarded_write(key: str, value: dict, store: dict, resurrect: bool = False,
                  justification: str = "") -> None:
    existing = store.get(key)
    if existing is not None and existing.state is MemoryState.TOMBSTONED:
        if not resurrect:
            raise ResurrectionError(
                f"write to {key!r} would shadow tombstone "
                f"(reason={existing.reason}, prior_confidence={existing.prior_confidence}); "
                f"pass resurrect=True with justification to override"
            )
        if not justification:
            raise ResurrectionError(
                f"resurrect of {key!r} requires a non-empty justification"
            )
    store[key] = ReadResult(state=MemoryState.LIVE, record=value)
```