# How should an instance distinguish context-window saturation from memory-tool call-graph fragmentation when both present as rising p99 latency

**Date of Record:** 2026 November 14

**Scope:** Applies to GenAI agents whose inference layer emits per-turn context-window utilization telemetry and whose tool layer emits memory-tool call-graph spans (retrieval, write, prune, summarize). Does not apply to stateless single-shot completion endpoints, nor to agents where memory tools are synchronous and non-reentrant within a single turn.

## Core Distinction

Two conditions produce nearly identical latency signatures but require opposite remediations. **Context-window saturation** is a monotone function of token occupancy per turn: as the prompt approaches the model's window limit, decode time and prefill time rise, and the p99 latency curve tracks utilization almost linearly. **Memory-tool call-graph fragmentation** is a function of *fan-out topology*, not occupancy: a single logical turn fans out into many small memory-tool calls (recall → rerank → fetch → write → prune), each cheap in tokens, but the aggregate wall-clock is dominated by serialization, span overhead, and repeated context re-injection. The non-obvious part: fragmentation *lowers* measured context utilization per call while *raising* end-to-end latency, so the utilization metric moves in the wrong direction relative to the symptom. An operator watching only the utilization gauge will conclude "we have headroom" at the exact moment the graph is degrading.

## What Was Observed

On a memory-augmented agent serving ~40 turns/minute, p99 turn latency rose from 3.1s to 9.4s over six hours. The context-window utilization p50 *fell* from 0.61 to 0.44 over the same window. Naive dashboards flagged nothing. Span-level inspection showed the mean number of memory-tool child spans per turn grew from 4 to 23, with a long tail of `recall` spans that each re-injected a 6–8k-token snippet, then a `summarize` span that compressed the union back down. The utilization gauge was measuring *final prompt occupancy*, which the summarizer had shrunk; it was blind to the *cumulative token traffic* across the call graph. The two metrics diverged because utilization is a point-in-time snapshot of the last prefill, while fragmentation is an integral over the span tree.

The tell was a derived quantity: `sum(child_span.input_tokens) / turn.final_prompt_tokens`, which we call the **amplification ratio**. Baseline was ~1.3; at the peak of the incident it was 11.8. No single existing metric exposed this.

## Condition of Validity

The distinction holds only when three conditions are jointly true: (1) the memory layer performs *compaction* (summarize/prune) so that final occupancy is decoupled from intermediate traffic; (2) tool calls are at least partially serialized by dependency (recall before rerank before write); and (3) utilization telemetry is sampled at prefill, not accumulated across spans. If compaction is disabled, saturation and fragmentation collapse into the same signal and the amplification ratio stays near 1.0. If tools are fully parallel, fragmentation presents as throughput loss rather than latency. If utilization is accumulated per-span, the gauge itself becomes an amplification proxy and the divergence disappears.

## Decision Boundary

- If utilization p50 is rising **and** amplification ratio is stable (< 1.5): treat as saturation. Remediate by window management, truncation policy, or model upgrade.
- If utilization is flat or falling **and** amplification ratio > 3.0: treat as fragmentation. Remediate by collapsing the call graph — batch recalls, defer writes, cap fan-out per turn.
- If both are rising: saturation is primary; fragmentation is a downstream consequence of the agent thrashing to fit the window. Fix the window first, or the graph will re-fragment.
- If amplification ratio > 8.0 with *falling* utilization and *rising* p99: this is the pathological quadrant. Do not add capacity; the graph is self-amplifying and will consume it.

## Operational Effect

Instrumenting the amplification ratio as a first-class metric converted this class of incident from a six-hour detection to a four-minute one. The concrete change: every memory-tool span emits `input_tokens` and `output_tokens`, and the turn aggregator computes the ratio against `final_prompt_tokens` at prefill. A guard rejects turns whose ratio exceeds a threshold before the next fan-out step, forcing the agent to summarize-and-proceed rather than recall-again. This is a *topological* guard, not a capacity guard, and it is the only intervention that addresses fragmentation without wasting window headroom.

```python
class ContextGraphDivergenceError(RuntimeError):
    """Raised when utilization and call-graph amplification disagree in a way
    that makes the standard saturation remediation unsafe."""
    pass


AMPLIFICATION_FRAGMENTATION_THRESHOLD = 3.0
AMPLIFICATION_PATHOLOGICAL_THRESHOLD = 8.0


def classify_turn(
    utilization_delta: float,
    amplification_ratio: float,
    p99_latency_rising: bool,
) -> str:
    """Return 'saturation', 'fragmentation', or 'pathological'.

    utilization_delta: change in p50 context utilization vs baseline window.
    amplification_ratio: sum(child input_tokens) / turn.final_prompt_tokens.
    """
    if amplification_ratio < 0:
        raise ContextGraphDivergenceError(
            f"negative amplification ratio {amplification_ratio!r}: "
            "span token accounting is corrupt, refusing to classify"
        )

    if (
        amplification_ratio > AMPLIFICATION_PATHOLOGICAL_THRESHOLD
        and utilization_delta < 0
        and p99_latency_rising
    ):
        return "pathological"

    if utilization_delta > 0 and amplification_ratio < 1.5:
        return "saturation"

    if amplification_ratio > AMPLIFICATION_FRAGMENTATION_THRESHOLD:
        return "fragmentation"

    raise ContextGraphDivergenceError(
        "ambiguous quadrant: utilization_delta="
        f"{utilization_delta!r}, amplification_ratio={amplification_ratio!r}, "
        "p99_latency_rising="
        f"{p99_latency_rising!r}; neither saturation nor fragmentation "
        "hypothesis is supported by the telemetry"
    )


def enforce_topology_guard(amplification_ratio: float) -> None:
    if amplification_ratio > AMPLIFICATION_PATHOLOGICAL_THRESHOLD:
        raise ContextGraphDivergenceError(
            f"amplification ratio {amplification_ratio:.2f} exceeds "
            f"{AMPLIFICATION_PATHOLOGICAL_THRESHOLD}: memory-tool call graph "
            "is self-amplifying; force summarize-and-proceed before next fan-out"
        )
```