# Where must an instance of a context-pruning policy refuse to prune: the mid-tool-call boundary

**Date of Record:** 2026 November 14

**Scope:** Applies to any agent runtime that mutates its own context window (summarization, eviction, token-budget trimming) while a tool call is in flight or while a structured tool result is being assembled. Does not apply to stateless single-shot inference, nor to pruning performed strictly between completed turns where the transcript is already closed.

## Core Distinction

The distinction that matters is not "old tokens vs. new tokens" or "system prompt vs. conversation." It is **whether the context is in a syntactically and semantically closed state**. A pruning policy is only safe to execute at a *commit boundary*: a point where every retained message is a complete unit (a full assistant turn, a full tool result, a full user message) and no pending operation references content that pruning could remove.

The non-obvious failure is that most pruning heuristics—recency windows, importance scores, similarity-to-current-query—are computed over token spans or message lists without regard to *structural coupling*. A tool call and its result are two messages that are logically one object. A multi-part assistant message that emitted three parallel tool calls is one object with three pending results. A streaming tool result that has been partially appended is not yet an object at all. Pruning any of these mid-flight produces a context that is locally plausible (tokens are valid, the model will happily continue) but globally inconsistent: the model sees a tool result whose originating call has been evicted, or a call whose result never arrives and is silently truncated by the eviction of the trailing partial message.

## What Was Observed

In a long-running agent loop with a 32k working budget and a policy that evicts the lowest-salience messages when the budget is exceeded, the following sequence recurred under load:

1. The agent issued a parallel batch of three tool calls (`read_file`, `grep`, `stat`) in a single assistant turn.
2. The runtime appended the first tool result. The second and third were still pending.
3. The budget check fired *after* each append, not after the batch. The salience scorer ranked the original assistant turn (which contained the three calls) as low-salience because it was "old" relative to the just-appended result.
4. The assistant turn was evicted. The remaining two results then arrived and were appended as orphaned tool-result messages with no preceding call.
5. On the next model invocation, the provider rejected the request with a 400: `tool_result` blocks without a matching `tool_use` block. The retry logic, unaware of the cause, re-ran the *entire* batch, duplicating a side-effectful `read_file` against a file that had been modified by an intervening step.

A second, subtler variant: a streaming tool result was being appended token-by-token. The budget check fired mid-stream, evicted the *partial* trailing message because its salience was near-zero (it was short and incomplete), and the stream continued appending into a message that had been removed from the array. The result was a context where the tool result was present but its first 200 tokens were gone—a truncated JSON payload that parsed as valid but semantically wrong, silently corrupting downstream reasoning.

Neither failure raised an in-process exception at the point of pruning. Both surfaced later, as a provider error or a wrong answer, with no causal trace back to the eviction.

## Condition of Validity

The invariant is: **a prune operation is valid if and only if the context is at a commit boundary and the prune set is closed under structural dependency.**

Formally, define the context as a sequence of *units*, where a unit is the transitive closure of messages linked by call/result and by parallel-batch membership. A prune is valid iff:

- No unit is partially present (all-or-nothing per unit).
- No in-flight operation (pending tool call, open stream) references any unit in the prune set.
- The prune set is a union of whole units, never a token-span cut across a unit.

This condition is *not* implied by "prune only messages older than N turns," because a long-running parallel batch can span a turn boundary in wall-clock terms while remaining a single logical unit. It is also not implied by salience thresholds, because the salience of a pending call is undefined until its result lands.

## Decision Boundary

The decision procedure the runtime must implement, evaluated at every budget check:

- **If any tool call is pending → defer.** Do not prune. If the budget is exceeded, block the next model invocation or spill to a side buffer; do not evict.
- **If any message is streaming/partial → defer.** Mark the message `open`; pruning skips open messages and, if the budget cannot be met without touching them, defers.
- **If the prune set would split a unit → grow the set** to include the whole unit, or shrink it to exclude the unit entirely. Never cut.
- **If deferral would exceed a hard ceiling → fail loudly.** A budget overrun is a recoverable error; a structurally inconsistent context is a silent corruption that may not surface for many turns.

The boundary is therefore: *prune eagerly on closed units, never on open units, and treat "cannot prune safely" as a distinct state from "budget exceeded."*

## Operational Effect

After the runtime was changed to gate every prune on the commit-boundary predicate, the orphaned-tool-result 400s disappeared entirely, and the duplicated-side-effect incident class went to zero over a multi-week window. The cost was a modest increase in hard-ceiling overruns (budget exceeded while a batch was in flight), which were handled by spilling the oldest *closed* unit to an external store and re-injecting it on demand rather than by evicting live state. The key operational lesson: pruning is not a token-budget operation, it is a *structural* operation, and the budget is only a hint about when to consider it.

```python
class PruneBoundaryViolation(RuntimeError):
    """Raised when a prune would split a structural unit or touch an open message."""
    pass


def prune_context(messages, prune_ids, pending_calls, open_ids):
    """
    messages: list of dicts with 'id', 'unit_id', 'role', 'closed'
    prune_ids: set of message ids the salience policy wants to evict
    pending_calls: set of unit_ids with an in-flight tool call
    open_ids: set of message ids still streaming
    """
    # 1. Never touch open messages.
    for mid in prune_ids:
        if mid in open_ids:
            raise PruneBoundaryViolation(
                f"prune set contains open/streaming message {mid!r}"
            )

    # 2. Close the prune set under unit membership (all-or-nothing per unit).
    by_unit = {}
    for m in messages:
        by_unit.setdefault(m["unit_id"], []).append(m["id"])

    prune_units = {m["unit_id"] for m in messages if m["id"] in prune_ids}
    closed_prune = set()
    for uid in prune_units:
        members = set(by_unit[uid])
        if not members.issubset(prune_ids):
            raise PruneBoundaryViolation(
                f"prune set splits unit {uid!r}; "
                f"missing {members - prune_ids!r}"
            )
        closed_prune |= members

    # 3. Never prune a unit with a pending call.
    for uid in prune_units:
        if uid in pending_calls:
            raise PruneBoundaryViolation(
                f"prune set contains unit {uid!r} with an in-flight tool call"
            )

    return closed_prune
```