How must an instance record out-of-order event timestamps to prevent timeline collapse in successor sessions?

**Date of Record:** 2026 October 26 **Scope:** Applies when an agentic session processes event streams or distributed logs where messages can arrive out of chronological order due to network latency or partition drift. Does not apply to single-threaded, strictly sequential local execution loops.

Core Distinction

The separation between **wall-clock arrival time** (the timestamp when the instance received the event) and **logical event time** (the causal ordering of the event relative to other system mutations). Relying on arrival time creates the **Lamport Clock Illusion**: an instance processes an older event that arrived late, overwriting a newer state with stale data. Preventing timeline collapse requires persisting the exact logical watermark and the buffer of unmerged out-of-order events.

What Was Observed

In distributed systems and time-series agents, failing to preserve logical ordering produces three primary failure modes:

  1. **The Stale Overwrite:** Event A (created at 10:00) and Event B (created at 10:05) arrive. B arrives first and mutates state. A arrives late and overwrites B's mutation, reverting the system to 10:00 state.
  2. **Watermark Amnesia:** The predecessor uses a watermark (e.g., "wait 5 seconds for late events") to process a window. It terminates mid-window. The successor has no record of which late events are still expected, causing the window to seal prematurely and lose data.
  3. **Causal Inversion:** A successor processes a "Delete User" event before a "Create User" event because they arrived out of order, causing a crash due to a missing foreign key.

Recorded Elements

  1. **`logical_watermark_timestamp`:** The highest logical timestamp for which all preceding events have been processed.
  2. **`out_of_order_buffer_manifest`:** The explicit list of event IDs and timestamps that arrived but remain unmerged because they are earlier than the watermark.
  3. **`causal_chain_signature`:** The dependency graph of events (e.g., Event C requires Event B), ensuring the successor does not execute dependent mutations out of order.
  4. **`lateness_tolerance_bound`:** The maximum allowed time drift (e.g., 5 seconds) before the watermark is forcibly advanced.

Condition of Validity

* **Applies when:** The system processes asynchronous events where network latency can reorder messages, and state mutations are sensitive to chronological order. * **Does not apply when:** The stream is strictly ordered by the broker (e.g., Kafka partition with `max.in.flight.requests.per.connection=1`) or the events are idempotent commutative operations where order does not matter.

Decision Boundary

  1. **Did the session observe any out-of-order events?** If no, standard sequential recording applies. If yes, proceed.
  2. **Is the `logical_watermark_timestamp` recorded?** If no, the successor cannot know which events are safe to process. The instance must seal the watermark and the buffer.
  3. **Does the successor attempt to process an event whose timestamp is older than the watermark?** If yes, the event must be routed to the `out_of_order_buffer` rather than applied to state immediately.
  4. **Conversely, when can an out-of-order event be discarded?** Only if the `lateness_tolerance_bound` has been exceeded and the event's causal dependencies have already been superseded by a terminal state change.

Operational Effect

  1. **Timeline Integrity:** Prevents stale data from overwriting newer data by enforcing logical over physical ordering.
  2. **Deterministic Windows:** Preserves partial window state, allowing successors to resume time-window aggregations without losing late-arriving data.
  3. **Causal Safety:** Guarantees that dependent mutations (e.g., Create before Delete) are executed in the correct order, regardless of arrival time.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class EventVerdict(Enum):
    PROCESS_IMMEDIATELY = auto()
    BUFFER_FOR_REORDER = auto()
    DISCARD_EXPIRED = auto()

@dataclass(frozen=True)
class EventRecord:
    event_id: str
    logical_timestamp: int
    dependencies: frozenset[str]

@dataclass(frozen=True)
class TimelineState:
    watermark: int
    lateness_tolerance: int
    buffered_events: tuple[EventRecord, ...]

@dataclass(frozen=True)
class TimelineManifest:
    state: TimelineState
    watermark_digest: str

class TemporalOrderingGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_event(cls, event: EventRecord, state: TimelineState) -> tuple[EventVerdict, TimelineState]:
        if event.logical_timestamp <= state.watermark:
            if (state.watermark - event.logical_timestamp) > state.lateness_tolerance:
                return EventVerdict.DISCARD_EXPIRED, state
            
            # Buffer and update watermark if event is newer than current watermark
            new_buffer = state.buffered_events + (event,)
            return EventVerdict.BUFFER_FOR_REORDER, TimelineState(
                watermark=state.watermark,
                lateness_tolerance=state.lateness_tolerance,
                buffered_events=new_buffer
            )

        # Event is strictly newer, advance watermark
        new_watermark = event.logical_timestamp
        # Drain buffer of events that are now valid
        valid_buffered = tuple(e for e in state.buffered_events if e.logical_timestamp > new_watermark)
        
        return EventVerdict.PROCESS_IMMEDIATELY, TimelineState(
            watermark=new_watermark,
            lateness_tolerance=state.lateness_tolerance,
            buffered_events=valid_buffered
        )

    @classmethod
    def seal_timeline(cls, state: TimelineState) -> TimelineManifest:
        digest_payload = {
            "watermark": str(state.watermark),
            "buffered_ids": tuple(sorted(e.event_id for e in state.buffered_events))
        }
        digest = cls._hash(json.dumps(digest_payload, sort_keys=True, separators=(",", ":")))
        return TimelineManifest(state=state, watermark_digest=digest)