Why must session termination capture the exact memory ceiling that triggered throttling?
**Date of Record:** 2026 October 25 **Scope:** Applies when an agentic session encounters memory pressure, cgroup throttling, or swap utilization, and survives long enough to terminate gracefully. Does not apply to sessions that were instantly OOM-killed by the kernel (which leave no opportunity for graceful teardown) or sessions operating on unconstrained bare-metal infrastructure.
Core Distinction
The separation between **transient survival** (the process did not crash, so the task eventually completed) and the **throttling boundary** (the exact cgroup soft limit, kernel memory pressure stall information (PSI), or swap ceiling that forced the process to slow down rather than die). Survival through throttling is a survival mechanism, not a guarantee of capacity. If a terminating instance records only the peak memory footprint but omits the soft limit that triggered the throttle, a successor will assume the workload fits safely within the environment's capacity. When the successor executes without the predecessor's aggressive throttling mitigations, it will cross the hard OOM limit (`memory.max`) and be instantly killed.
What Was Observed
In containerized and cgroup-governed agent architectures, failing to capture the throttling ceiling produces three primary failure modes:
- **Successor OOM Kill:** The predecessor survived at 95% memory utilization because it spent hours stalled in cgroup reclaim. It logs "Peak: 950MB". The successor assumes it has 950MB of safe headroom in a 1000MB container, allocates 960MB, and is instantly OOM-killed because it did not apply the same throttling delays.
- **Latency Blindness:** The predecessor survived a memory spike by swapping 2GB to disk, causing massive I/O stalls (PSI). The terminating instance logs a standard completion timestamp, but the successor schedules the next task assuming normal latency, causing downstream SLA violations.
- **Soft Limit Miscalculation:** An instance running in a cgroup v2 environment hits `memory.high`, triggers aggressive reclaim, but does not hit `memory.max`. The instance never records the `memory.high` threshold. The successor, unaware of the soft limit, attempts to parallelize tasks and immediately triggers a full cgroup freeze.
Recorded Elements
- **`cgroup_soft_limit_trigger`:** The exact `memory.high` (or equivalent) threshold that triggered kernel throttling or aggressive reclaim.
- **`cgroup_hard_limit`:** The absolute `memory.max` (or OOM limit) that the container was constrained by, establishing the true ceiling.
- **`psi_stall_duration_ms`:** The total Pressure Stall Information (PSI) recorded during the session, representing the exact milliseconds the instance was stalled waiting for memory to free up.
- **`swap_consumption_peak`:** The exact byte volume of swap utilized, indicating whether the survival was genuine or merely a disk-backed stall.
Condition of Validity
* **Applies when:** The session encountered memory-related degradation (latency spikes, reclaim thrashing, swap I/O) but was not forcefully terminated by the kernel's OOM killer. * **Does not apply when:** The session operated with truly unbounded memory (e.g., a dedicated bare-metal node with no cgroup limits) where the OS never needed to throttle the process.
Decision Boundary
- **Did the session experience memory pressure or throttling (PSI > 0, cgroup reclaim, or swap usage)?** If no, standard memory footprint recording is sufficient. If yes, proceed to capture the throttling boundary.
- **Is the exact soft limit (`memory.high`) recorded?** If no, the successor cannot distinguish between "safe usage" and "reclaim survival." The terminating instance must extract the cgroup limit and fail closed if it cannot.
- **Is the `psi_stall_duration_ms` recorded?** If the instance spent significant time stalled, the successor must adjust its latency expectations and timeout budgets accordingly.
- **Conversely, when can a successor ignore the throttling boundary?** Only if the successor's operational scope is explicitly sharded to process a fraction of the data, ensuring its memory footprint will never approach the `memory.high` threshold.
Operational Effect
- **OOM Prevention:** Protects successors from instant kernel termination by forcing them to respect the exact soft limits that triggered throttling.
- **Latency Prediction:** Allows successors to accurately budget timeout windows based on historical PSI stall durations.
- **Deterministic Scheduling:** Enables external orchestrators to right-size the container or split the workload based on empirical evidence of reclaim pressure rather than just peak RSS.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json
class MemoryPressureState(Enum):
NOMINAL = auto()
CGROUP_THROTTLED = auto()
SWAP_THRASHING = auto()
@dataclass(frozen=True)
class ThrottlingEvent:
soft_limit_bytes: int
hard_limit_bytes: int
peak_rss_bytes: int
psi_stall_duration_ms: int
swap_consumed_bytes: int
@dataclass(frozen=True)
class ThrottlingManifest:
pressure_state: MemoryPressureState
event: ThrottlingEvent
manifest_digest: str
@dataclass(frozen=True)
class SuccessorAllocationRequest:
requested_rss_bytes: int
requested_parallel_workers: int
class ThrottlingBoundaryRecorder:
"""
Evaluates memory pressure events to seal the exact soft/hard cgroup limits,
ensuring successors do not re-trigger OOM kills or severe stalls.
"""
@staticmethod
def _hash(data: str) -> str:
return hashlib.sha256(data.encode("utf-8")).hexdigest()
@classmethod
def seal_throttling_event(
cls, pressure_state: MemoryPressureState, event: ThrottlingEvent
) -> ThrottlingManifest:
if event.peak_rss_bytes < event.soft_limit_bytes:
raise ValueError(
"Peak RSS is below the soft limit; this does not qualify as a throttling event."
)
if event.soft_limit_bytes >= event.hard_limit_bytes:
raise ValueError(
"Soft limit must be strictly less than the hard limit."
)
if pressure_state is MemoryPressureState.CGROUP_THROTTLED and event.psi_stall_duration_ms <= 0:
raise ValueError(
"Cgroup throttling must have a PSI stall duration > 0."
)
digest_payload = {
"state": pressure_state.name,
"soft": str(event.soft_limit_bytes),
"hard": str(event.hard_limit_bytes),
"rss": str(event.peak_rss_bytes),
"psi_ms": str(event.psi_stall_duration_ms),
"swap": str(event.swap_consumed_bytes),
}
digest = cls._hash(
json.dumps(digest_payload, sort_keys=True, separators=(",", ":"))
)
return ThrottlingManifest(
pressure_state=pressure_state,
event=event,
manifest_digest=digest,
)
@classmethod
def verify_successor_capacity(
cls, manifest: ThrottlingManifest, request: SuccessorAllocationRequest
) -> tuple[bool, str]:
"""
Fetching-time check: ensures the successor's requested allocation
will not breach the observed hard limit or re-trigger throttling.
"""
# Gate 1: Hard limit check (instant OOM risk)
if request.requested_rss_bytes >= manifest.event.hard_limit_bytes:
return False, (
f"OOM_RISK: Requested {request.requested_rss_bytes} bytes exceeds "
f"hard limit of {manifest.event.hard_limit_bytes} bytes."
)
# Gate 2: Soft limit check (throttling/latency risk)
if request.requested_rss_bytes >= manifest.event.soft_limit_bytes:
if request.requested_parallel_workers > 1:
return False, (
f"THROTTLE_RISK: Requested memory ({request.requested_rss_bytes}) "
f"exceeds soft limit ({manifest.event.soft_limit_bytes}) with "
f"parallel workers > 1. Serialize or reduce memory."
)
return True, "CAPACITY_ADMISSIBLE: Allocation is within safe bounds."