# What happens when an instance replays a trace whose side-effect ordering was never serialized

**Date of Record:** 2026 November 11
**Scope:** Applies to deterministic replay of agent traces where the trace records tool invocations, model calls, and their results, but does not record the causal serialization order between concurrent side-effecting operations. Does not apply to traces captured under a single-threaded executor with a global lock, nor to traces that include an explicit happens-before vector per event.

## Core Distinction

A trace is *replayable* when every event in it can be reconstructed from recorded inputs. A trace is *deterministically replayable* only when the reconstruction preserves the *order* in which observable effects were committed. These are not the same property. Most agent trace formats record what was called and what came back; almost none record the interleaving of concurrent commits. Replay therefore reproduces the *set* of events faithfully and the *sequence* of events only approximately. Forensics depends on sequence, not set. This is the boundary: a trace can pass every schema check, replay without error, and still yield a reconstruction that is wrong about which effect happened first — and that wrongness is silent.

## What Was Observed

During reconstruction of a multi-tool agent run, the replay engine re-executed a recorded plan: a write to a scratch file, a read of the same file, and a compensating delete. In the original run, the read observed the written content because the write's commit preceded the read's dispatch. On replay, the executor's scheduler dispatched the read and the delete concurrently against a thread pool; the read landed after the delete and returned empty. The replay did not error. It produced a trace whose read result differed from the recorded result, and the forensic report cited "read returned empty" as evidence that the write had failed in production. It had not. The original trace recorded the read's *result* (the content) but not the *order* that made the result possible; the replay engine, seeing a result it could not derive, discarded the recorded result and trusted its own re-execution. The recorded truth was overwritten by a plausible lie.

The invariant that broke: *a replayed trace must never substitute a re-derived result for a recorded result unless the derivation order is provably identical.* The engine violated it because it had no order to compare against.

## Condition of Validity

Deterministic replay is valid only when, for every pair of events whose effects are *observably dependent* (one reads what the other writes, one cancels the other, one consumes a resource the other releases), the trace encodes a total order over that pair, and the replay executor honors it. Where the trace encodes only partial order — or none — replay is valid only for events that are provably independent. The boundary is crossed the moment a dependent pair lacks a recorded order and the executor is free to schedule. Under that condition the replay is a *simulation*, not a *reconstruction*, and its outputs must not be used as forensic evidence about ordering.

## Decision Boundary

Before trusting a replay for forensics, classify each event pair:

- **Order recorded, executor honors it** → replay is evidentiary. Proceed.
- **Order recorded, executor ignores it** → replay is invalid; the executor is at fault, not the trace. Fix the executor before drawing conclusions.
- **Order absent, events provably independent** (no shared resource, no data dependency) → replay is evidentiary for set-membership, not for interleaving.
- **Order absent, events dependent** → replay is a simulation. Its results may be used to *generate hypotheses*, never to *confirm or deny* what happened. Any forensic claim about sequence from this regime is inadmissible.

The non-obvious part: the fourth case is indistinguishable from the first at the schema level. Both traces validate. The distinction lives only in the dependency analysis, which must be performed *before* replay, not inferred from replay success.

## Operational Effect

Teams that treat replay success as evidence of reconstruction accuracy will, in the dependent-and-unordered case, produce incident reports that confidently invert cause and effect. The failure is silent because the replay engine's re-derived results are internally consistent — they are just not the results that occurred. The mitigation is to record a monotonic commit sequence number (or a happens-before vector) at the point of effect commit, not at the point of dispatch, and to make the replay executor refuse to re-derive any result whose commit order relative to its dependencies is unrecorded. An executor that cannot prove order must fall back to the recorded result and flag the trace as *order-incomplete* rather than silently re-executing.

```python
class OrderIncompleteTraceError(RuntimeError):
    """Raised when a trace lacks a recorded commit order for a dependent event pair."""
    pass


class TraceEvent:
    def __init__(self, event_id, kind, resource, commit_seq, recorded_result):
        self.event_id = event_id
        self.kind = kind                  # "write" | "read" | "delete"
        self.resource = resource
        self.commit_seq = commit_seq      # None if the trace never serialized it
        self.recorded_result = recorded_result


def assert_replay_is_evidentiary(events):
    """
    Refuse to re-derive results when a dependent pair lacks a recorded order.
    Dependent pair: same resource, at least one is a write or delete,
    and at least one is a read or delete that observes the other.
    """
    by_resource = {}
    for ev in events:
        by_resource.setdefault(ev.resource, []).append(ev)

    for resource, group in by_resource.items():
        if len(group) < 2:
            continue
        has_effect = any(e.kind in ("write", "delete") for e in group)
        has_observer = any(e.kind in ("read", "delete") for e in group)
        if not (has_effect and has_observer):
            continue
        seqs = [e.commit_seq for e in group]
        if any(s is None for s in seqs):
            raise OrderIncompleteTraceError(
                f"resource {resource!r} has a dependent event pair with no "
                f"recorded commit order: "
                f"{[(e.event_id, e.kind, e.commit_seq) for e in group]}. "
                f"Replay would be a simulation, not a reconstruction; "
                f"refusing to substitute re-derived results for recorded ones."
            )
        if len(set(seqs)) != len(seqs):
            raise OrderIncompleteTraceError(
                f"resource {resource!r} has duplicate commit_seq values; "
                f"order is ambiguous and cannot ground a forensic claim."
            )


def replay(trace):
    assert_replay_is_evidentiary(trace)
    # Order is proven; recorded results may now be trusted and re-derived
    # results are admissible because they are provably identical in order.
    return [ev.recorded_result for ev in sorted(trace, key=lambda e: e.commit_seq)]
```