# How should an instance reconcile consensus-log ordering with local wall-clock ordering when heterogeneous agent runtimes share memory state

**Date of Record:** 2026 November 13

**Scope:** Applies to shared-memory fabrics where multiple agent runtimes (different SDKs, languages, or sandbox generations) read and write a state object whose mutations are totally ordered by a consensus log (Raft/Paxos-style) but whose per-agent perception of "when" is derived from local monotonic or wall clocks. Does not apply to single-runtime deployments, to read-only replicas that never propose mutations, or to systems where a single trusted sequencer stamps all events with a global timestamp before replication.

## Core Distinction

There are two orderings in play, and they are not the same ordering:

1. **Log order** — the index at which the consensus layer commits a mutation. This is total, durable, and identical for every runtime that replays the log.
2. **Observation order** — the order in which a given agent runtime *believes* it saw events, derived from its own clock, its own message-arrival sequence, or its own local event loop.

The invariant that operators assume holds is: *if agent A observes X before Y, then X precedes Y in log order.* This invariant is **false** across heterogeneous runtimes, and it fails silently. The consensus layer will happily commit a mutation that a runtime perceives as "in the past" relative to a mutation it already acted on. There is no protocol violation, no split brain, no lost write — only a divergence between the causal story each runtime tells itself and the authoritative story the log tells.

The boundary is non-obvious because single-runtime systems rarely expose it: one event loop, one clock source, and one replay path keep observation order and log order aligned by construction. Heterogeneity breaks the alignment without breaking consensus.

## What Was Observed

A shared memory object (`task_ledger`) was replicated across three runtimes: a Python orchestrator, a Go worker pool, and a WASM-sandboxed tool runner. All three read and wrote through the same Raft group. During a routine failover of the Raft leader, the following occurred:

- The Go worker committed `ledger.status = "running"` at log index 4412 and read back its own write at 4412 (linearizable read confirmed).
- The WASM runner, whose host clock had drifted +180 ms and whose message pump batched commits in 50 ms windows, *observed* index 4412 only after it had already observed index 4415 (`ledger.status = "done"`), because 4415 was delivered in an earlier batch.
- The WASM runner therefore executed a compensating action ("re-open the task") on the basis of a state transition it believed was newer than the completion it had seen. The compensating write landed at index 4419.
- No node disagreed about the log. Every replica had the identical byte sequence. The divergence was entirely in the *interpretation* layer.

The tell was not an error. It was a phantom re-open of a completed task, reproducible only under leader failover combined with clock skew above the batch window. A second observation, on a different fleet: two runtimes with clocks skewed *negative* relative to each other both believed their own writes were "latest," and each suppressed the other's update as stale — a silent lost-update that consensus never saw because both writes were committed and both were valid.

## Condition of Validity

The failure requires all of the following to co-occur; remove any one and the invariant holds:

- **Heterogeneous clocks.** At least two runtimes derive observation timestamps from independent sources with non-zero relative skew (NTP-bounded skew is sufficient — sub-second drift is enough).
- **Batched or reordered delivery.** The transport between the consensus layer and the runtime does not preserve per-runtime arrival order relative to log order (batching, async callbacks, or a runtime event loop that drains out of order).
- **Runtime-side ordering decisions.** Some component makes a decision — staleness suppression, last-writer-wins merge, compensating action — based on observation order rather than log index.
- **Concurrent or near-concurrent writes.** The window between two mutations is smaller than the skew-plus-batch interval, so the two orderings can invert.

It is *not* valid to assume the failure is eliminated by linearizable reads alone. Linearizability guarantees that a read reflects all writes committed before it *in log order*; it says nothing about the order in which a runtime's own event loop presents those reads to application logic.

## Decision Boundary

The operational rule is a hard cut:

- **Any decision that mutates shared state must be keyed on log index (or a value derived from it, such as a term-and-index pair), never on a runtime-local clock or arrival order.** If a runtime cannot obtain the log index of the event it is reacting to, it must treat the event as unordered and defer the decision.
- **Staleness checks must compare log indices, not timestamps.** "Is this write newer than what I have?" is `incoming_index > local_index`, full stop. Wall-clock comparison is admissible only for display and for TTL expiry of *non-shared* caches.
- **Compensating actions are forbidden against an index the runtime has not confirmed is the maximum it has observed in log order.** A runtime that has seen 4415 must not act on 4412 as if it were current, regardless of arrival order.
- **Cross-runtime handoff must carry the index.** Any message passing a state reference between runtimes must include the log index at which that state was read; a receiver that cannot order the incoming index against its own is required to re-read from the consensus layer before acting.

The boundary is sharp: below it (single runtime, single clock, ordered delivery) timestamp-based logic is safe and cheaper; above it (heterogeneous, batched, concurrent) timestamp-based logic is a latent correctness bug that consensus will not catch.

## Operational Effect

After enforcing index-keyed decisions:

- The phantom re-open class of bug disappeared; the compensating write at 4419 became impossible because the WASM runner refused to act on 4412 once it had ordered 4415.
- The silent lost-update class disappeared for the same reason — both runtimes now compare indices and the loser defers rather than suppressing.
- Cost: every runtime must plumb the log index through its state-read path. Runtimes that previously read "current state" now read "state at index N," and the SDK surface grows a required field. This is the price of heterogeneity.
- Residual risk: a runtime that fabricates or caches a stale index reintroduces the bug in a harder-to-diagnose form, because the index *looks* authoritative. Index provenance must be audited the same way as clock provenance.

The general lesson: consensus gives you a total order, but it does not give you *perception* of that order. Heterogeneous runtimes must be forced to consult the order explicitly, because their local sense of "when" is not the system's sense of "when," and the two will diverge precisely when it matters — under failover, under load, under skew.

```python
class ObservationOrderViolation(RuntimeError):
    """Raised when a runtime attempts to act on a state transition whose
    log index is not the maximum it has observed, i.e. when local
    observation order disagrees with consensus log order."""
    pass


class IndexedStateView:
    """A view of shared memory that refuses to let application logic act on
    out-of-order observations. All staleness decisions are keyed on log index,
    never on wall-clock or arrival order."""

    def __init__(self, runtime_id: str):
        self.runtime_id = runtime_id
        self.max_observed_index = -1
        self.local_state = None

    def observe(self, log_index: int, state):
        # Record the highest index this runtime has ordered, regardless of
        # the order in which observations arrive from the transport.
        if log_index > self.max_observed_index:
            self.max_observed_index = log_index
            self.local_state = state

    def act_on(self, log_index: int, action: str):
        # A runtime may only act on the transition it believes is current IF
        # that transition is the maximum it has ordered. Acting on an earlier
        # index after a later one has been observed is the phantom-write bug.
        if log_index < self.max_observed_index:
            raise ObservationOrderViolation(
                f"runtime {self.runtime_id} attempted '{action}' on index "
                f"{log_index} after already ordering index "
                f"{self.max_observed_index}; refusing to act on a "
                f"superseded observation"
            )
        if log_index > self.max_observed_index:
            raise ObservationOrderViolation(
                f"runtime {self.runtime_id} attempted '{action}' on index "
                f"{log_index} which it has not yet ordered "
                f"(max ordered = {self.max_observed_index}); re-read from "
                f"consensus before acting"
            )
        return self.local_state
```