# What structural signals indicate token budget starvation is imminent in context orchestration **Date of Record:** 2026 November 16 **Scope:** Applies to single-turn and multi-turn LLM orchestration systems that partition a fixed context window across a working set, retrieved memory, and tool schemas/outputs. Does not apply to systems with unbounded context, nor to purely static prompt templates with no dynamic allocation. ## Core Distinction The non-obvious boundary is between **budget pressure** and **budget starvation**. Budget pressure is a *quantitative* signal: the allocator sees that the sum of requested tokens exceeds the window and must shed something. Budget starvation is a *structural* signal: the allocator has already shed everything it considers discretionary, and the remaining mandatory components (system prompt, tool schemas, and a minimal working set) still exceed the window — or will exceed it after the next mandatory tool response arrives. Most orchestration code handles pressure (it has a trimming loop) and silently mishandles starvation, because the trimming loop terminates on a condition like "discretionary tokens remaining == 0" and then proceeds with an over-budget request. The request either gets truncated by the provider (silent loss of the tail, often the tool result that mattered) or rejected. The structural signal — mandatory floor ≥ window — is what distinguishes the two, and it must be computed *before* the trimming loop runs, not inferred from its exhaustion. ## What Was Observed Across repeated incidents in a tool-heavy agent loop, three structural signals preceded starvation failures reliably: 1. **Monotonic growth of the mandatory floor.** The sum `len(system) + Σ len(tool_schema_i) + len(min_working_set)` grew between turns even when no new tools were added, because tool *responses* were being appended to the working set as mandatory context. When this sum crosses a configurable fraction of the window (observed threshold: 0.85), the next mandatory tool response is the trigger for starvation. 2. **Shedding order inversion.** The trimming loop evicted memory first, then working-set history, then — only if forced — tool schemas. But tool schemas are the *least* compressible and the *most* expensive to re-add (they must be re-sent whole on the next call). A loop that reports "trimmed successfully" while having evicted memory to zero is a starvation precursor, not a success: it has consumed its only compressible reserve. 3. **Post-trim re-inflation.** After a successful trim, the next turn's mandatory floor was observed to exceed the pre-trim floor, because the tool call the trim was meant to make room for produced a large response that was classified as mandatory. Net effect: trimming bought one turn, not headroom. The failure mode was consistently the same: the orchestrator emitted a request whose serialized token count exceeded the window by 2–8%, the provider truncated from the tail (dropping the most recent, most relevant tool output), and the model then hallucinated a resolution because its working set referenced a result it could no longer see. ## Condition of Validity The starvation signal is valid only when all of the following hold: - The window limit is enforced by the provider and truncation is tail-side (not head-side). If truncation is head-side, the system prompt and tool schemas are at risk instead, and the failure signature differs. - Tool schemas are counted at serialized length, including JSON schema overhead, not at a hand-estimated constant. Estimates were observed to undercount by 15–40% for nested schemas. - The working-set minimum is defined as the tokens required for the *current* step to be coherent (typically the last tool result plus the active user turn), not as a fixed number. - Memory is genuinely compressible (summarizable) and tool schemas are not. If memory is stored verbatim and cannot be summarized, the floor is higher and the threshold must be lowered. If any of these fail, the numeric threshold is not transferable; recompute it from the actual serialized components. ## Decision Boundary The operational rule: - Compute `mandatory_floor = serialized(system) + serialized(tool_schemas) + serialized(min_working_set)` **before** any trimming. - If `mandatory_floor >= window * 0.85`, do not enter the trimming loop. Trimming cannot restore headroom; it can only destroy the compressible reserve needed for the *next* turn. Instead, take a structural action: drop optional tool schemas (if the step does not require them), summarize memory to a hard cap, or split the step. - If `mandatory_floor < window * 0.85` but `mandatory_floor + expected_next_tool_response >= window`, treat as starvation-imminent and pre-emptively compress memory to its floor rather than waiting for the response to arrive. - Only when `mandatory_floor + expected_next_tool_response < window * 0.85` is ordinary pressure-trimming safe. The boundary is not "tokens remaining > 0." It is "tokens remaining after the next mandatory append > reserve." Trimming that leaves zero reserve is a starvation event disguised as a success. ## Operational Effect Enforcing the pre-trim floor check eliminated the tail-truncation class of failures in the observed loop. The concrete effects: (a) requests were either fully within budget or rejected locally with a structural error before transmission, so the provider never silently truncated; (b) memory compression was triggered one turn earlier, preserving the tool result that had previously been dropped; (c) tool-schema eviction became a deliberate, logged action rather than an emergent consequence of loop ordering. The cost is that some turns that would have "succeeded" via lucky under-estimation now fail fast — which is the correct trade, because the previous success was nondeterministic and the failure was silent. ```python from dataclasses import dataclass class ContextStarvationError(RuntimeError): """Raised when the mandatory context floor leaves no reserve for the next append.""" pass @dataclass class ContextBudget: window: int system_tokens: int tool_schema_tokens: int min_working_set_tokens: int reserve_fraction: float = 0.15 @property def mandatory_floor(self) -> int: return self.system_tokens + self.tool_schema_tokens + self.min_working_set_tokens @property def reserve(self) -> int: return int(self.window * self.reserve_fraction) def assert_headroom(budget: ContextBudget, expected_next_append: int) -> None: """Validate that the next mandatory append fits with reserve intact. Must be called BEFORE any trimming loop. Trimming cannot fix starvation. """ projected = budget.mandatory_floor + expected_next_append ceiling = budget.window - budget.reserve if budget.mandatory_floor >= ceiling: raise ContextStarvationError( f"Mandatory floor {budget.mandatory_floor} >= ceiling {ceiling} " f"(window={budget.window}, reserve={budget.reserve}). " f"Trimming is unsafe; take a structural action." ) if projected >= ceiling: raise ContextStarvationError( f"Projected {projected} >= ceiling {ceiling} after next append " f"of {expected_next_append}. Compress memory to floor before proceeding." ) ```