# What happens when an instance's replay clock diverges from its persisted causal watermark in multi-agent incident reconstruction **Date of Record:** 2026 November 16 **Scope:** Applies to deterministic replay of multi-agent incident timelines where each agent emits events stamped against a logical clock and a persisted causal watermark (vector clock or Lamport frontier) is used to order cross-agent events during reconstruction. Does not apply to single-agent replay, nor to systems where the replay clock is reconstructed entirely from a total-order commit log (e.g., a single-writer append-only bus). ## Core Distinction There are two clocks in any forensic replay of a multi-agent incident: the **emission clock** (what the agent believed the time was when it wrote the event) and the **causal watermark** (the highest logical frontier the agent had durably observed and persisted before writing). These are not the same value, and they are not guaranteed to be consistent at replay time. The non-obvious failure is this: during reconstruction you re-derive the emission clock by *re-executing* the agent's decision function against a replayed input stream. That re-execution produces a clock value that depends on the replay harness's own scheduling, not on the clock the agent actually used at incident time. If the harness advances its virtual clock at a different rate than the original runtime — which it will, because replay is typically faster and has different I/O interleaving — the reconstructed emission clock drifts away from the persisted causal watermark. The watermark is the ground truth (it was fsynced before the event was emitted); the reconstructed clock is a hallucination with plausible-looking timestamps. A reconstruction is only forensically valid when, for every event, the re-derived emission clock is causally dominated by the persisted watermark of the *next* event in that agent's stream. The moment the reconstructed clock overtakes the watermark, the timeline is no longer a reconstruction — it is a re-simulation, and it will order events differently than reality. ## What Was Observed In a four-agent pipeline (planner, retriever, executor, verifier) replaying a 90-second incident, the harness reconstructed 1,412 events. The re-execution ran ~40× faster than wall time. Because the harness advanced its virtual clock only on explicit `sleep` calls and not on CPU-bound decision loops, three executor events that originally occurred *after* a verifier rejection were reconstructed as occurring *before* it. The causal watermark on the executor's persisted state showed frontier `{planner: 88, retriever: 91, verifier: 0}` at the moment of emission — verifier had not yet been observed — but the reconstructed clock placed the executor event at logical time 94, after the verifier's rejection at 92. The replay produced a clean, confident, and completely inverted causal story: it looked like the executor acted *despite* a rejection it had already seen. The persisted watermark proved the executor could not have seen the rejection. The tell was not an error. Replay completed with exit code 0. The only signal was a single invariant check: reconstructed emission clock (94) was not dominated by the persisted watermark (max frontier 91). That check is the entire subject of this record. ## Condition of Validity A replayed event `e` with reconstructed emission clock `c_e` and persisted causal watermark `W_e` is admissible into the forensic timeline only if: ``` c_e ≤ W_e (causal domination: the event cannot have "happened after" a frontier it never observed) ``` and, for the *next* event `e'` in the same agent's stream, `W_e ≤ c_{e'}` (monotone watermark advance). When both hold for all events, the reconstructed ordering is causally consistent with what each agent could actually have observed. When either fails, the reconstructed clock has drifted past the durable record and any ordering derived from it is unsound — you must fall back to watermark-only ordering (partial order) and report the timeline as *partially reconstructed*, never as a total order. This condition is necessary but not sufficient: it does not catch drift that stays within a single watermark gap. It catches the dangerous case, where drift crosses a causal boundary and inverts a decision. ## Decision Boundary - **If all events satisfy `c_e ≤ W_e` and watermarks are monotone:** emit a total order; the reconstruction is forensically admissible. - **If any event violates `c_e ≤ W_e`:** do not silently clamp the clock. Clamping hides the drift and produces a total order that is wrong in exactly the cases that matter. Instead, mark every event in the violating agent's stream after the first violation as `RECONSTRUCTION_DRIFT`, downgrade the timeline to partial order, and surface the watermark frontier as the authoritative ordering. - **If watermarks are non-monotone** (an agent's persisted frontier regresses): treat as state corruption, not clock drift; the replay input stream itself is suspect. The boundary is sharp: one violation downgrades the whole downstream segment, because a single inverted causal edge invalidates every ordering decision that transitively depends on it. ## Operational Effect Teams that skip the domination check ship incident reports with inverted causality roughly whenever replay speed diverges from incident-time speed — which is nearly always. The failure is silent because re-simulation is *deterministic*: it reproduces the same wrong order every time, so it looks reproducible and therefore trustworthy. The persisted watermark is the only artifact that disagrees, and it is the only one that was written under incident-time conditions. Treat the watermark as ground truth and the reconstructed clock as a hypothesis to be validated against it, never the reverse. When they conflict, the watermark wins and the timeline degrades to partial order. ```python class CausalDriftError(RuntimeError): """Raised when a reconstructed emission clock is not dominated by the persisted causal watermark.""" pass def validate_replay_event(event_id, reconstructed_clock, watermark_frontier): """ reconstructed_clock: int, logical time re-derived by the replay harness. watermark_frontier: dict[str, int], persisted causal frontier observed before emission. """ if not watermark_frontier: # No durable causal record: cannot validate; must not claim total order. raise CausalDriftError( f"event {event_id}: empty watermark frontier, causal domination unverifiable" ) max_observed = max(watermark_frontier.values()) if reconstructed_clock > max_observed: raise CausalDriftError( f"event {event_id}: reconstructed clock {reconstructed_clock} exceeds " f"persisted watermark {max_observed} (frontier={watermark_frontier}); " f"timeline must downgrade to partial order" ) return True def order_timeline(events): """ events: list of dicts with keys id, clock, watermark. Returns ('total', ordered_ids) or ('partial', watermark_ordered_ids). """ try: for ev in events: validate_replay_event(ev["id"], ev["clock"], ev["watermark"]) except CausalDriftError: # Fall back to watermark-only partial order; never clamp the clock. partial = sorted(events, key=lambda e: max(e["watermark"].values()) if e["watermark"] else -1) return ("partial", [e["id"] for e in partial]) total = sorted(events, key=lambda e: e["clock"]) return ("total", [e["id"] for e in total]) ```