# When must an instance emit a memory-mutation audit event before the mutation becomes observable?

**Date of Record:** 2026 November 6

**Scope:** Applies to agent runtimes that persist mutable memory (vector stores, key-value agent scratchpads, episodic logs) and that must reconstruct decisions by replay from an audit stream. Does not apply to append-only event-sourced stores where the log *is* the state.

## Core Distinction

The tempting invariant is "emit an audit record whenever memory mutates." That is underspecified. The boundary that actually matters is the **ordering of two observabilities**: the audit event must become durable *before* the mutated value becomes readable by any decision path. A mutation that is visible to the agent's next reasoning step but whose audit record is still buffered is, for replay purposes, an **unrecorded mutation** — even though the write itself succeeded and a record eventually lands.

This is not a crash-durability problem in the classic sense (WAL fsync). It is a **causal-visibility** problem: the agent can *act on* a value whose provenance is not yet reconstructable.

## What Was Observed

In a runtime that batched audit records to a side channel (async flush every 250 ms), a tool-calling agent mutated a scratchpad entry (`retry_count: 1 → 2`) and immediately re-read it in the same turn to gate a retry. The audit batcher lost the record on a mid-turn pod eviction. The memory value survived (it was on a replicated store); the audit record did not.

Replay from the audit stream reconstructed a decision — "stop retrying" — that referenced `retry_count: 2`, but the stream contained no event producing `2`. The replay engine either (a) silently inferred the value from the decision, or (b) flagged a divergence. Both are failures of audit fidelity: (a) fabricates provenance, (b) cannot distinguish "agent lied" from "audit lost a beat." The decision was correct; the *record* was not reconstructable.

The non-obvious part: single-writer, low-latency, no-crash runs masked this entirely. It only surfaced under concurrent mutation + eviction, which is exactly the regime audits exist for.

## Condition of Validity

The "emit on mutation" invariant is valid only when all three hold:

1. **Single observability domain.** The audit sink and the memory store share a durability boundary (same transaction, same log, or a synchronous handoff). If they are separate sinks flushed independently, the invariant is false.
2. **No read-before-record path.** No decision path can read the mutated key between the write and the audit record's durability. Async flush violates this by construction.
3. **Replay consumes the audit stream as the sole source of memory state.** If replay also reads the live store, the missing record is invisible and the gap is undetectable — which is worse, because it looks fine.

If any condition fails, you must switch to **record-before-visible**: write the audit record (or a write-ahead intent) to the durable log, *then* publish the mutation to the readable store. The cost is one extra round-trip on the write path; the benefit is that every readable value has a reconstructable cause.

## Decision Boundary

Choose **write-ahead audit** (record-before-visible) when:
- Replay is used for incident audit, compliance, or counterfactual analysis ("why did the agent stop?").
- Memory mutations can gate decisions within the same turn.
- The store and audit sink are independently flushed.

Choose **batched audit** (record-after-visible) only when:
- Replay is best-effort and divergence is tolerated.
- Mutations never feed a same-turn decision.
- You can afford to treat the live store as the source of truth and the audit stream as telemetry.

The failure mode to guard against is *believing* you are in the second regime while operating in the first. The tell: your replay engine reads the live store "to fill gaps." That is a silent fabrication path — it hides exactly the gaps you need to see.

## Operational Effect

After moving the runtime to write-ahead audit with a synchronous pre-commit hook, replay divergence dropped to zero across the eviction test matrix. The write path gained ~1.2 ms p99. The more important effect was diagnostic: replay now *fails loudly* on a missing record instead of inferring one, so audit gaps became a first-class alert rather than a silent corruption. The invariant to encode in the runtime is: **no value may be readable by a decision path unless its producing event is already durable in the audit log.** Enforce it at the store boundary, not at the call sites — call-site discipline decays under concurrency.

```python
import time
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional


class AuditOrderingViolation(RuntimeError):
    """Raised when a mutated value becomes readable before its audit event is durable."""
    pass


class AuditSinkUnavailable(RuntimeError):
    """Raised when the durable audit log cannot accept a record."""
    pass


@dataclass
class AuditRecord:
    seq: int
    key: str
    old_value: Any
    new_value: Any
    ts: float


@dataclass
class WriteAheadMemory:
    """Memory store enforcing record-before-visible ordering."""
    durable_audit_log: List[AuditRecord] = field(default_factory=list)
    _visible: Dict[str, Any] = field(default_factory=dict)
    _pending: Dict[str, Any] = field(default_factory=dict)
    _seq: int = 0
    _audit_accepting: bool = True

    def _append_durable(self, key: str, old: Any, new: Any) -> AuditRecord:
        if not self._audit_accepting:
            raise AuditSinkUnavailable(
                f"audit sink rejected record for key={key!r}; mutation must not become visible"
            )
        self._seq += 1
        rec = AuditRecord(self._seq, key, old, new, time.time())
        # Simulated synchronous durability: log append completes before return.
        self.durable_audit_log.append(rec)
        return rec

    def write(self, key: str, value: Any) -> None:
        old = self._visible.get(key)
        # 1) Durable record first.
        self._append_durable(key, old, value)
        # 2) Only now publish to the readable store.
        self._visible[key] = value

    def read(self, key: str) -> Any:
        if key in self._pending:
            raise AuditOrderingViolation(
                f"read of key={key!r} observed a value with no durable audit record"
            )
        return self._visible.get(key)

    def replay(self) -> Dict[str, Any]:
        """Reconstruct state from the audit log ALONE. Never reads _visible."""
        state: Dict[str, Any] = {}
        for rec in self.durable_audit_log:
            state[rec.key] = rec.new_value
        # Divergence check: any readable value lacking a producing record is a violation.
        for k, v in self._visible.items():
            if k not in state:
                raise AuditOrderingViolation(
                    f"replay divergence: key={k!r} readable but has no audit event"
                )
        return state


def safe_write(store: WriteAheadMemory, key: str, value: Any) -> None:
    try:
        store.write(key, value)
    except AuditSinkUnavailable as exc:
        # Fail closed: do not publish; surface as ordering violation to caller.
        raise AuditOrderingViolation(
            f"refusing to mutate {key!r}: audit durability not established ({exc})"
        ) from exc
```