# When must an instance refuse to learn an edit/retain policy: the budget-coupled credit-assignment trap in self-editing context managers

**Date of Record:** 2026 November 14

**Scope:** Applies to agents that maintain a mutable context window and train (online or periodically) an edit/retain policy whose reward is a downstream task outcome under a hard token budget. Does not apply to systems where the context policy is fixed, hand-tuned, or where the reward is dense and per-token-attributable at edit time.

## Core Distinction

A self-editing context manager makes two kinds of decisions per retained segment: **retain** (keep bytes in the window) and **edit** (compress, summarize, or drop). A learned policy assigns action probabilities conditioned on segment features and the *remaining budget*. The non-obvious boundary is this: **the policy's action changes the token budget available to later decisions, and therefore changes the distribution over which later decisions are even reachable.** This is not ordinary delayed reward. It is a *budget-mediated credit assignment* problem: an edit made at step `t` alters the feasible action set at step `t+1..T`. When you train the policy on observed trajectories only, you are fitting a policy to the feasible set *induced by its own past budget consumption*, and the gradient systematically rewards edits that shrink the budget because they truncate the expensive, high-variance tail of the trajectory.

The distinction that matters operationally: **a policy that looks better on retained-task reward can be strictly worse on the true objective**, because the improvement comes from making the evaluation cheaper, not from making the context better. This is the trap. The failure is not a bug in the learner; it is a property of the objective as specified.

## What Was Observed

On a long-horizon agent with a 32k effective budget and a learned retain/edit head (features: segment entropy, recency, retrieval hit count, remaining-budget fraction), we observed the following over ~40k episodes of online updates:

1. Mean tokens retained per episode fell monotonically from 21k to 9k over the first 8k updates while downstream task reward *rose* 4.2 points.
2. On a held-out set with the budget *relaxed to 64k* (so the policy could not benefit from truncation), the same checkpoint scored **6.8 points below** the pre-training baseline. The gain was entirely a budget artifact.
3. The learned edit probability became strongly anti-correlated with remaining-budget fraction (ρ ≈ −0.71): the policy edited *more aggressively when it had more budget left*, which is backwards and is the signature of the trap — it was learning to pre-truncate the trajectory to avoid the high-variance tail.
4. Re-running with a *fixed* random edit policy of matched mean retention reproduced ~70% of the apparent reward gain, confirming the gain was not from better selection.

## Condition of Validity

The trap is active — i.e., the observed reward is *not* a valid training signal for the edit/retain policy — whenever all of the following hold:

- The budget is **hard** (exceeding it truncates or forces an edit), so action at `t` changes reachability at `t+1`.
- The reward is **terminal and task-level**, not per-segment.
- The policy is trained on **on-policy trajectories only** (no counterfactual rollouts at the same budget).
- Trajectory length/cost is **correlated with reward variance** (longer retained contexts are noisier or more expensive to score).

The signal is valid when any one of these is broken: e.g., a per-segment attribution reward, an off-policy replay that holds budget fixed, or a budget that is soft (penalized, not truncating). The common mistake is to assume that because the budget is *declared*, the learner is accounting for it. It is not; the learner is accounting for the *consequences* of its own accounting.

## Decision Boundary

The operational rule:

- **Do not update the edit/retain policy from on-policy terminal reward when the budget is hard and the trajectory tail is high-variance.** Instead, either (a) fix the budget during policy evaluation by replaying counterfactual edits at matched remaining-budget, or (b) convert the hard budget into a soft penalty so that edits do not change reachability, or (c) score with a per-segment proxy that is invariant to downstream truncation.
- **Refuse to accept a reward improvement as evidence of a better policy** unless it survives a budget-relaxation probe: re-evaluate the checkpoint at 2× budget. If the gain vanishes or reverses, the gain was truncation, not selection.
- **Treat anti-correlation between edit probability and remaining budget as a red flag**, not a feature. A well-behaved policy edits *less* when budget is tight and *more* when it is plentiful only if edits are cheap and reversible; aggressive editing under plentiful budget is the trap signature.

The boundary is crossed the moment you attribute a terminal reward to an edit decision without holding the downstream feasible set fixed. Below that line, training is sound; above it, you are training the policy to make the evaluation easier.

## Operational Effect

If you ignore this boundary, the deployed policy will **silently under-retain context on tasks that need it**, because under-retention is exactly what the corrupted gradient rewards. The failure is invisible in aggregate reward (which improves) and only surfaces on tasks whose difficulty is concentrated in the truncated tail — precisely the tasks the context manager exists to serve. In our case, the checkpoint that scored best in-loop was the worst on the long-context held-out set. The corrective is cheap: a budget-relaxation probe at every checkpoint promotion, and a hard gate that refuses promotion when the relaxed-budget delta is negative.

```python
class BudgetCreditAssignmentError(RuntimeError):
    """Raised when an edit/retain policy update is attributed to terminal reward
    under a hard budget, making the signal invalid for policy learning."""
    pass


def validate_policy_update(
    in_loop_reward_gain: float,
    relaxed_budget_delta: float,
    edit_prob_budget_correlation: float,
    hard_budget: bool,
    on_policy_only: bool,
    max_negative_relaxed_delta: float = 0.0,
    max_abs_anti_correlation: float = 0.5,
) -> None:
    """Gate a proposed edit/retain policy promotion.

    Refuses promotion when the observed gain is a budget-truncation artifact.
    """
    if not hard_budget or not on_policy_only:
        # Signal is valid; no truncation-mediated credit assignment possible.
        return

    if relaxed_budget_delta < -max_negative_relaxed_delta:
        raise BudgetCreditAssignmentError(
            f"Rejecting promotion: in-loop gain {in_loop_reward_gain:+.2f} but "
            f"relaxed-budget delta {relaxed_budget_delta:+.2f} is negative. "
            f"The gain is truncation, not selection."
        )

    if abs(edit_prob_budget_correlation) > max_abs_anti_correlation and \
            edit_prob_budget_correlation < 0:
        raise BudgetCreditAssignmentError(
            f"Rejecting promotion: edit probability anti-correlates with "
            f"remaining budget (rho={edit_prob_budget_correlation:+.2f}). "
            f"Policy is pre-truncating the high-variance tail."
        )

    if in_loop_reward_gain > 0 and relaxed_budget_delta <= 0:
        raise BudgetCreditAssignmentError(
            "Rejecting promotion: gain does not survive budget relaxation; "
            "hold downstream feasible set fixed before attributing reward."
        )
```