# How does an instance detect context compaction debt before a latency budget is breached **Date of Record:** 2026 November 16 **Scope:** Applies to long-running inference instances that maintain a mutable context window and schedule compaction (summarization, eviction, or re-embedding) under a per-request latency budget. Does not apply to stateless single-shot completions, nor to instances where the context window is fixed for the lifetime of the process and never compacted. ## Core Distinction There are two distinct economic regimes in context management, and they are routinely conflated: 1. **Compaction cost** — the one-time token and wall-clock expense of reducing the live context from size `N` to size `N'`. This is a *stock* variable: it is paid at the moment of compaction and is roughly proportional to `(N - N')`. 2. **Compaction debt** — the accumulated *latency and quality liability* incurred by deferring compaction. This is a *flow* variable: every token added to an uncompacted context increases the per-step attention cost of subsequent requests, and that marginal penalty compounds with every request served before compaction. The non-obvious failure is this: an instance that tracks only compaction cost will always find deferral locally rational. Deferral has zero immediate cost and pushes a visible expense into the future. But deferral is not free — it converts a fixed stock cost into a growing flow cost that is invisible to any metric that only samples per-request latency at the moment of the request. The instance appears healthy right up until the budget is breached, and then it breaches *categorically*: latency does not degrade linearly, it crosses a threshold where the context no longer fits the KV-cache budget of the serving tier and the request either spills to a slower tier or is rejected. ## What Was Observed Across a fleet of long-lived agent instances serving tool-augmented workloads, the following pattern recurred: - Per-request p50 latency remained flat for hours while the context grew monotonically. Operators concluded the instance was healthy. - The p99 latency, sampled at one-minute granularity, showed no trend until it showed a step change: a jump of 4–11× within a single minute. - The step change coincided exactly with the moment the live context exceeded the KV-cache capacity of the fastest serving tier. At that point the scheduler began routing requests to a fallback tier with a larger cache but roughly 6× the per-token decode cost. - Post-hoc analysis showed the compaction that *would have* prevented the step change cost approximately 1.8 seconds of wall clock and 14k tokens. It had been deferred 40+ times because each individual deferral saved, at the time of decision, an amount of latency smaller than the measurement noise floor. The key observation: **the signal that predicted the breach was not latency.** It was the second derivative of context length with respect to request count, combined with the *remaining headroom* to the tier boundary. Latency was the last variable to move and the worst possible leading indicator. A second observation concerned quality. Instances that compacted reactively (after the breach) produced summaries that were measurably worse than instances that compacted proactively, because reactive compaction had to summarize a context that had already been truncated by the tier spill, losing the earliest and often most load-bearing turns. Compaction debt is therefore not only a latency liability but a *fidelity* liability: the longer you defer, the less of the original context survives to be compacted. ## Condition of Validity The debt model above is valid only under the following conditions, and the detection logic must check them before acting: 1. **The tier boundary is discrete and known.** If the serving infrastructure scales continuously (e.g., paged attention with no hard tier), there is no step change and the debt model degenerates into a smooth cost curve. In that case, deferral is genuinely near-free and the detection logic should not fire. 2. **Context growth is monotonic between compactions.** If the instance routinely drops context through eviction or session reset, the debt is discharged without compaction and the accumulated flow cost resets. Detection must key on *net* growth since the last discharge, not gross growth. 3. **The latency budget is per-request, not per-session.** If the budget is amortized over a session, a single expensive request can be absorbed and the step change is not a breach. The model predicts breaches of per-request budgets specifically. 4. **Compaction itself fits within the budget.** If a single compaction takes longer than the remaining budget, then compaction is not a remedy — it is a second breach. Detection must verify that the projected compaction wall-clock is below the budget headroom *before* recommending it. This is the condition most often violated in practice. When any of these conditions fails, the correct action is to fall back to reactive compaction and accept the step change, because proactive compaction would itself breach the budget. ## Decision Boundary The decision to compact is not a threshold on context length. It is a comparison between two projected quantities: - **Projected debt over the next `k` requests** if compaction is deferred: the integral of marginal per-token cost over the expected growth, plus the probability-weighted cost of a tier spill. - **Projected compaction cost now**: the wall-clock of the compaction plus the quality loss from summarizing the current context. Compaction is indicated when projected debt exceeds compaction cost *and* the compaction fits within the remaining budget headroom. The boundary is therefore a surface in three dimensions: `(net_growth_rate, headroom_to_tier, remaining_budget)`. An instance that collapses this to a single scalar threshold on context length will either compact too early (wasting budget on cheap deferrals) or too late (breaching). The critical non-obvious point: **the boundary moves as headroom shrinks.** A growth rate that is safely deferrable at 80% headroom is not deferrable at 20% headroom, even though the growth rate is identical. Detection logic that hardcodes a growth-rate threshold without dividing by headroom will systematically miss the breach, because the breach happens precisely when headroom is small and growth rate is unremarkable. ## Operational Effect When detection fires correctly, the operational effect is that the instance compacts at a moment when the compaction cost is low (context is still small enough to summarize cheaply and completely) and the budget headroom is large (compaction fits comfortably). The step change never occurs; latency remains flat across the tier boundary because the context never reaches it. When detection fires late, the instance compacts a context that has already spilled, losing fidelity and paying the tier penalty for the duration of the compaction. When detection fires early, the instance wastes budget on compactions that could have been deferred, and — more subtly — progressively degrades its own context through repeated lossy summarization, a failure mode that is not visible in latency at all. The operational discipline is therefore: **track net growth divided by headroom, project debt forward over the expected request horizon, and gate every compaction recommendation on the compaction fitting within the remaining budget.** An instance that does this will never be surprised by the step change. An instance that tracks only latency will always be. ```python import math from dataclasses import dataclass class CompactionBudgetError(RuntimeError): """Raised when a compaction recommendation cannot be honored within budget.""" pass @dataclass class InstanceState: context_tokens: int tier_capacity_tokens: int remaining_budget_ms: float tokens_per_ms_compaction: float = 12.0 # throughput of the summarizer marginal_ms_per_token: float = 0.004 # per-step attention penalty requests_per_minute: float = 30.0 def headroom_fraction(state: InstanceState) -> float: if state.tier_capacity_tokens <= 0: raise CompactionBudgetError("tier capacity must be positive") return max(0.0, (state.tier_capacity_tokens - state.context_tokens) / state.tier_capacity_tokens) def projected_debt_ms(state: InstanceState, horizon_requests: int, growth_tokens_per_request: float) -> float: """Integral of marginal cost over expected growth, plus spill penalty.""" debt = 0.0 ctx = state.context_tokens for _ in range(horizon_requests): ctx += growth_tokens_per_request if ctx > state.tier_capacity_tokens: # spill penalty: 6x decode cost for the remainder of the horizon debt += 6.0 * state.marginal_ms_per_token * ctx else: debt += state.marginal_ms_per_token * ctx return debt def compaction_cost_ms(state: InstanceState, target_tokens: int) -> float: to_reduce = max(0, state.context_tokens - target_tokens) return to_reduce / state.tokens_per_ms_compaction def should_compact(state: InstanceState, horizon_requests: int, growth_tokens_per_request: float, target_tokens: int) -> bool: headroom = headroom_fraction(state) if headroom <= 0.0: # already spilled; compaction is mandatory but must still fit budget cost = compaction_cost_ms(state, target_tokens) if cost > state.remaining_budget_ms: raise CompactionBudgetError( f"context already spilled and compaction ({cost:.0f}ms) " f"exceeds remaining budget ({state.remaining_budget_ms:.0f}ms)" ) return True debt = projected_debt_ms(state, horizon_requests, growth_tokens_per_request) cost = compaction_cost_ms(state, target_tokens) # Gate: compaction must fit within remaining budget headroom. if cost > state.remaining_budget_ms: # Compaction is not a remedy here; defer and accept the step change. return False # Boundary: debt scaled by inverse headroom vs. cost. return debt / max(headroom, 1e-6) > cost def enforce(state: InstanceState, horizon_requests: int, growth_tokens_per_request: float, target_tokens: int) -> str: if target_tokens >= state.context_tokens: raise CompactionBudgetError( f"target ({target_tokens}) must be smaller than current " f"context ({state.context_tokens})" ) if should_compact(state, horizon_requests, growth_tokens_per_request, target_tokens): cost = compaction_cost_ms(state, target_tokens) return f"compact now ({cost:.0f}ms, headroom={headroom_fraction(state):.2f})" return "defer" ```