# Why does an instance of the agent observability pipeline silently drop memory-mutation spans when the safety verdict emitter backpressures

**Date of Record:** 2026 November 12

**Scope:** Applies to agent observability pipelines that emit three correlated span classes — reasoning/trace spans, memory-mutation events, and safety verdicts — into a shared collector with a bounded per-trace buffer. Does not apply to pipelines where safety verdicts are emitted on an independent transport (e.g., a separate sidecar socket) or where memory mutations are journaled to durable storage before emission.

## Core Distinction

The failure is not "the collector dropped spans." It is a *causal ordering* violation inside the per-trace buffer. The pipeline's trace assembler enforces a single ordering key per trace: `(trace_id, monotonic_seq)`. Memory mutations and safety verdicts are emitted by different producers with different latency profiles. The safety verdict emitter is synchronous with the policy model and can stall for tens to hundreds of milliseconds under load; the memory-mutation emitter is asynchronous and cheap. When the safety verdict emitter stalls, the assembler's per-trace buffer fills with mutation spans whose `monotonic_seq` values are *ahead* of the not-yet-arrived verdict. On buffer pressure the assembler evicts from the tail — the oldest-arrived, highest-seq entries — which are precisely the mutation spans. The verdict eventually arrives, is committed, and the trace is closed as *complete* because the assembler's completeness predicate only checks for the terminal trace span and the verdict, not for mutation spans. The result is a trace that asserts "no memory mutation occurred before this verdict" when in fact mutations occurred and were discarded.

The non-obvious part: the pipeline reports `spans_emitted - spans_dropped = spans_committed` with zero error, and the mutation spans are not counted in `spans_dropped` because eviction from the assembler buffer is logged as `buffer_evict`, a distinct counter that dashboards do not alert on. Observability of the observability pipeline is the gap.

## What Was Observed

A replay of 40k agent episodes against a policy model pinned at 92% GPU utilization produced a divergence: 1,847 episodes where the recorded trace contained zero memory-mutation spans but the agent's own scratchpad (captured separately) showed `write_memory` calls. Correlating by `trace_id`:

- All 1,847 traces had a safety verdict span whose `emit_latency_ms` exceeded the 95th percentile (>180 ms).
- All had at least one mutation span with `monotonic_seq` greater than the verdict's `monotonic_seq` — i.e., mutations were emitted *before* the verdict arrived in wall-clock terms but assigned *later* sequence numbers by the mutation producer's local clock.
- The assembler's `buffer_evict` counter was non-zero for every one of these traces; the `spans_dropped` counter was zero.
- The completeness predicate `has_terminal and has_verdict` returned true, so the trace was sealed.

Reduced reproducer: a synthetic producer that emits 50 mutation spans at 1 ms intervals and a verdict span after a 200 ms sleep, against an assembler buffer of capacity 32. Eviction begins at mutation span 33; the sealed trace reports `mutation_count=0`, `verdict=present`, `status=complete`.

## Condition of Validity

This boundary condition holds when and only when all of the following are true:

1. The assembler buffer is bounded and evicts on pressure rather than applying backpressure to producers.
2. The safety verdict emitter shares the assembler's transport with mutation spans (co-located producer, same gRPC stream or same in-process channel).
3. The completeness predicate does not require mutation spans — it treats them as optional annotations rather than required causal predecessors of the verdict.
4. Mutation spans and verdict spans derive `monotonic_seq` from producer-local clocks rather than a single sequencer.

If any one is false, the failure mode changes shape: (1) false → producers block and the agent stalls (visible, loud); (2) false → verdicts are independently durable and mutation eviction is recoverable from the verdict's side channel; (3) false → the trace is marked `incomplete` and the divergence surfaces as a data-quality alert; (4) false → ordering is consistent and eviction drops the *correct* (latest) spans, which is at least semantically coherent.

The condition is *not* valid for pipelines that use a single sequencer with backpressure, nor for pipelines that journal mutations to WAL before emission. Those have different, better-understood failure modes.

## Decision Boundary

The operational decision is whether to treat "verdict present, mutations absent" as evidence of no mutation or as evidence of pipeline loss. The boundary:

- If `buffer_evict > 0` for the trace → treat mutation absence as *unknown*, not *none*. The trace must be quarantined and the episode's safety verdict must not be used to close a memory-consistency audit.
- If `buffer_evict == 0` and `spans_dropped == 0` → mutation absence is credible; the agent genuinely did not mutate memory.
- If `buffer_evict > 0` and the verdict's `emit_latency_ms` is below the assembler's eviction threshold → the eviction is attributable to a different producer (reasoning spans) and mutation absence remains credible *only if* the mutation producer's own emission counter matches the assembler's received count.

The threshold is not a fixed latency; it is the assembler's `evict_pressure_window`, the wall-clock span over which the buffer must remain full to trigger tail eviction. In the observed deployment this was 40 ms; verdicts slower than 40 ms reliably triggered mutation eviction when concurrent mutation rate exceeded `capacity / window`.

## Operational Effect

Downstream, the sealed-but-lossy traces corrupt two classes of analysis: (a) memory-consistency audits that infer "no write preceded the safety verdict" and therefore under-report policy violations; (b) agent-training data that treats the trace as ground truth for credit assignment, teaching the agent that `write_memory` calls are sometimes no-ops. The second is the more insidious: it degrades the agent's learned model of its own memory semantics.

Mitigation, in order of preference: (1) make mutation spans a required predecessor in the completeness predicate — a trace without them is `incomplete`, not `complete`; (2) emit safety verdicts on an independent transport so they cannot backpressure mutation spans; (3) replace producer-local `monotonic_seq` with a single sequencer so eviction drops the semantically correct spans; (4) alert on `buffer_evict` at the trace level, not only in aggregate.

The following guard encodes the decision boundary as an executable check that fails closed on the ambiguous case.

```python
class AmbiguousTraceError(RuntimeError):
    """Raised when a trace's mutation absence cannot be distinguished from pipeline loss."""
    pass


REQUIRED_PREDECESSORS = ("terminal_span", "safety_verdict")


def classify_mutation_absence(trace: dict) -> str:
    """
    Returns 'none' or 'unknown'. Never returns 'none' when eviction is present
    and the verdict's latency could have triggered tail eviction.
    """
    evict = trace.get("buffer_evict", 0)
    dropped = trace.get("spans_dropped", 0)
    mutation_count = trace.get("mutation_span_count", 0)
    verdict_latency_ms = trace.get("verdict_emit_latency_ms", 0.0)
    evict_window_ms = trace.get("evict_pressure_window_ms", 40.0)

    if mutation_count > 0:
        return "present"

    if evict == 0 and dropped == 0:
        return "none"

    if evict > 0 and verdict_latency_ms >= evict_window_ms:
        # Eviction occurred and the verdict was slow enough to have caused it.
        # Mutation absence is indistinguishable from loss.
        raise AmbiguousTraceError(
            f"trace {trace.get('trace_id')}: mutation absence ambiguous "
            f"(buffer_evict={evict}, verdict_latency_ms={verdict_latency_ms:.1f}, "
            f"evict_window_ms={evict_window_ms:.1f}); "
            f"trace must be quarantined, not sealed as complete"
        )

    if evict > 0 and verdict_latency_ms < evict_window_ms:
        # Eviction attributable to another producer; require emission-count match.
        if trace.get("mutation_emitted_count", 0) != trace.get("mutation_received_count", 0):
            raise AmbiguousTraceError(
                f"trace {trace.get('trace_id')}: mutation emission/receipt mismatch "
                f"({trace.get('mutation_emitted_count')} emitted vs "
                f"{trace.get('mutation_received_count')} received) despite low verdict latency"
            )
        return "none"

    raise AmbiguousTraceError(
        f"trace {trace.get('trace_id')}: unclassified eviction state"
    )


def seal_trace(trace: dict) -> dict:
    for pred in REQUIRED_PREDECESSORS:
        if not trace.get(pred):
            raise AmbiguousTraceError(
                f"trace {trace.get('trace_id')}: missing required predecessor {pred!r}; "
                f"refusing to seal as complete"
            )
    verdict = classify_mutation_absence(trace)
    trace["mutation_absence_class"] = verdict
    trace["status"] = "complete" if verdict == "none" else "quarantined"
    return trace
```