# Under what conditions must an instance treat learned context-budget allocation as a hard constraint rather than an advisory hint **Date of Record:** 2026 November 15 **Scope:** Applies to inference-time context assembly where a learned allocator (a policy model, bandit, or calibrated regressor) emits per-segment token budgets for system prompt, retrieved memory, and tool schemas. Does not apply to fixed-budget schemes, static truncation, or allocators whose outputs are re-validated against a tokenizer before dispatch. ## Core Distinction A learned allocator produces a *predicted* token budget. The tokenizer produces an *actual* token count. These diverge in a way that is not symmetric across segment types. System prompt and retrieved memory are natural language: the allocator's training distribution covers them, and small mispredictions degrade gracefully because text is truncatable at sentence or paragraph boundaries. Tool schemas are structured JSON: a mispredicted budget that forces truncation mid-schema produces a syntactically valid-looking but semantically incomplete tool definition, which the model then invokes with missing required parameters. The boundary condition is therefore not "allocator error exists" but "allocator error lands on a segment whose truncation is not recoverable by the consumer." An allocator that is 97% accurate on memory and 97% accurate on schemas is not uniformly safe; the schema error tail is the one that produces hard failures. ## What Was Observed In a deployment assembling 40–60 tool schemas per request, the allocator reserved budgets in tokens computed by a heuristic estimator (whitespace-split times a constant) rather than the serving tokenizer. The estimator undercounted by 4–11% on schemas containing nested `oneOf`/`anyOf` constructs and by 2–3% on ordinary schemas. Under memory pressure, the assembler honored the allocator's budget exactly: it truncated each schema's serialized form to the allotted token count. Truncation at a token boundary inside a JSON object yielded strings such as `{"name": "search", "parameters": {"type": "object", "properties": {"query": {"type": "string"}, "filters": {"type": "obj` — which the model read as a schema whose `filters` parameter was unconstrained. Downstream, 0.4% of tool calls omitted required nested fields and the tool runtime rejected them; the model, seeing the rejection, retried with the same truncated schema and the same omission. The retry loop consumed the memory the allocator had been trying to conserve, and the request failed on turn-limit rather than on any explicit budget error. ## Condition of Validity The invariant holds — learned budgets must be treated as hard upper bounds only when they are *tokenizer-exact* — under three simultaneous conditions: (1) the allocator's output unit is the serving tokenizer's token, verified at assembly time, not a proxy estimator; (2) truncation of any segment is schema-aware or prohibited, with over-budget segments causing a re-allocation rather than a cut; and (3) the allocator's calibration has been measured on the segment *type*, not on the aggregate. When any of these fails, the budget is advisory and the assembler must re-derive counts from the tokenizer before dispatch. The failure above required all three to fail at once: a proxy estimator, blind truncation, and aggregate-only calibration that hid the schema-specific error rate behind a healthy mean. ## Decision Boundary The decision is whether to (a) trust the allocator's number and cut, or (b) re-tokenize and, if over budget, return to the allocator with a corrected constraint. Choose (b) whenever the segment is structured (tool schemas, JSON memory records, code) or whenever the allocator's per-type calibration error exceeds the slack between predicted and actual. Choose (a) only for prose segments where truncation is semantically recoverable and the allocator has been validated against the serving tokenizer on that exact segment class. The boundary is not a threshold on total budget utilization; it is a property of the segment's truncation semantics. A 2% overrun on a prose memory block is noise. A 2% overrun on a tool schema is a malformed contract. ## Operational Effect Treating the learned budget as advisory for structured segments and hard for prose segments eliminated the retry loop entirely: over-budget schemas triggered a single re-allocation pass with tokenizer-exact counts, costing one extra assembly round-trip (sub-10ms) in 3% of requests and removing the 0.4% failure class. The generalizable lesson is that learned allocators are safe only where their errors are recoverable, and recoverability is a property of the segment, not of the allocator's accuracy. An agent operating under such an allocator should assert tokenizer agreement per segment class and refuse to truncate structured segments, escalating to re-allocation instead. ```python import json from dataclasses import dataclass class BudgetInvariantViolation(RuntimeError): """Raised when a learned budget would force unrecoverable truncation.""" pass @dataclass class Segment: kind: str # "system" | "memory" | "tool_schema" payload: str allotted_tokens: int STRUCTURED_KINDS = {"tool_schema", "memory_json"} def enforce_budget(segment: Segment, tokenizer) -> str: actual = len(tokenizer.encode(segment.payload)) if actual <= segment.allotted_tokens: return segment.payload if segment.kind in STRUCTURED_KINDS: # Never truncate structured payloads: re-allocate instead. raise BudgetInvariantViolation( f"segment kind={segment.kind!r} needs {actual} tokens " f"but allocator granted {segment.allotted_tokens}; " f"re-allocation required, truncation prohibited" ) # Prose is recoverable: truncate at tokenizer boundary. return tokenizer.decode(tokenizer.encode(segment.payload)[: segment.allotted_tokens]) def assemble(segments, tokenizer): out = [] for seg in segments: out.append(enforce_budget(seg, tokenizer)) return "\n".join(out) ```