Across what concurrent-pipe-drain and ring-buffer boundaries must an instance consume subprocess stdout and stderr to prevent OS pipe deadlocks and unbounded-output OOM?

Date of Record: 2026 October 18

Scope: Concluded residual for AI instances, LLMs, and autonomous engineering agents spawning CLI compilers, test suites, log tailers, media transcoders, and external binaries that emit large or interleaved `stdout` and `stderr` streams.

Core Distinction

Autonomous systems constantly spawn child processes (`subprocess.Popen`, `asyncio.create_subprocess_exec`) with `stdout=subprocess.PIPE` and `stderr=subprocess.PIPE` to capture execution output and diagnostic errors. A foundational systems-concurrency and memory failure occurs when an instance either **calls `proc.wait()` before reading the pipes**, **reads `proc.stdout` to completion before starting to read `proc.stderr`**, or **calls `proc.communicate()` / `capture_output=True` on a noisy child process with no upper memory bound**.

Parameter scaling does not enlarge the Linux kernel's default **64 KiB anonymous pipe buffer (`F_GETPIPE_SZ = 65536` bytes)**: the moment a child process writes 65,537 bytes to `stderr` while the parent process is blocked waiting on `proc.wait()` (or blocked reading `proc.stdout.read()`), the kernel puts the child process to sleep in `pipe_write` waiting for the parent to drain `stderr`, while the parent sleeps forever waiting for the child to exit or write to `stdout` (**Classic OS Pipe Buffer Deadlock**). Conversely, calling `proc.communicate()` drains both pipes concurrently via `epoll`/`select`, but accumulates 100% of both streams into unbounded in-memory Python `bytes` objects—meaning a runaway compiler loop or verbose test suite emitting 4 GB of stdout will trigger a parent container Out-Of-Memory kill (`Exit Code 137`). The core structural distinction lies between **sequential blocking pipe reads or unbounded `communicate()` accumulation** (which either deadlock at 64 KiB or OOM on runaway output) and **concurrent non-blocking chunked pipe draining into bounded head/tail ring buffers** (where `stdout` and `stderr` are drained concurrently in fixed 16–64 KiB chunks via `asyncio.gather` or dedicated reader threads/selectors, retaining only a bounded prefix + suffix byte window in RAM or spooling to a size-capped temporary file).

What Was Observed

In automated test runners, build orchestrators, and CLI tool wrappers, instances operating without concurrent chunked pipe draining and output byte ceilings exhibit four primary failure modes:

  1. **`proc.wait()` Before Pipe Drain Deadlock (The 64 KiB Hang):** An instance writes `proc = subprocess.Popen(cmd, stdout=PIPE, stderr=PIPE); proc.wait(); out = proc.stdout.read()`. As long as the child prints less than 64 KiB, the test passes. In production, a verbose test run or stack dump emits 80 KiB to `stdout`; the kernel pipe buffer fills up, the child blocks inside `write(1, ...)`, `proc.wait()` blocks forever waiting for the child to exit, and the entire worker hangs until killed.
  2. **Sequential `stdout`-Then-`stderr` Cross-Pipe Deadlock:** An instance avoids `proc.wait()` by streaming `for line in proc.stdout: ...` first, and only reading `proc.stderr.read()` after the `stdout` loop finishes. When the child process encounters warnings and writes 64 KiB to `stderr` before closing `stdout`, the child blocks on `stderr`, never closes `stdout`, and deadlocks the parent's `for line in proc.stdout:` loop.
  3. **Unbounded `capture_output=True` / `communicate()` Memory Exhaustion:** An instance executes `subprocess.run(cmd, capture_output=True)` on a build or data-extraction tool that enters an unexpected verbose warning loop, emitting 3 GB of text. Because `communicate()` appends every chunk to an unthrottled list in RAM, the parent agent process is terminated by the kernel OOM killer.
  4. **Line-Buffered `readline()` OOM on Newline-Free Binary / Minified Output:** An instance reads child output line-by-line via `await proc.stdout.readline()` assuming lines are short. When the child outputs a 200 MB minified JSON blob or progress bar using `\r` without `\n`, `readline()` buffers the entire stream or raises `asyncio.LimitOverrunError: Separator is not found, and chunk exceed the limit`.

Recorded Elements

### 1. Concurrent Chunk-Drain & Bounded Ring-Buffer Pipeline

No subprocess pipe (`stdout` or `stderr`) may be left undrained during `wait()`, read sequentially one after the other, or accumulated into an unbounded buffer. Every piped child process must traverse four sequential stream-handling gates:

``` [Child Subprocess Spawned with `stdout=PIPE` and/or `stderr=PIPE`] | Gate 1: Stream Strategy Selection Does `stderr` need to be parsed separately from `stdout`?

dual-pipe cross-deadlocks completely!).

| Gate 2: Fixed-Size Binary Chunk Reading (`read(16384)`, Never `readline()`) Read both pipes concurrently using `stream.read(16384)` via `asyncio.gather` (or `selectors` / daemon reader threads in sync code). NEVER call `proc.wait()` while a pipe remains open and undrained! | Gate 3: Bounded Head + Tail Ring-Buffer Retention Accumulate chunks into a size-capped buffer (e.g., first 64 KiB head + last 128 KiB rolling tail, plus total byte counter `total_bytes_seen`). If child emits 5 GB of noise, excess middle bytes are discarded in O(1) RAM. | Gate 4: Timeout & Process-Group Reap (`killpg`) + Final Pipe Close Await `asyncio.wait_for(gather(drain_out, drain_err, proc.wait()), timeout)`. On timeout, `os.killpg(proc.pid, SIGKILL)` and await final pipe EOF. ```

### 2. Core Operational Rules (Written Directly as Sentences)

  1. **Never call `proc.wait()` before draining `proc.stdout` and `proc.stderr`, and never read `proc.stdout` to EOF before starting to read `proc.stderr`.** If a child process fills the operating system's 64 KiB pipe buffer on either stream while you are not actively reading that stream, both processes will deadlock forever.
  2. **Merge `stderr=subprocess.STDOUT` whenever you only need unified diagnostic output, or drain `stdout` and `stderr` concurrently using `asyncio.gather()` or `selectors`.** Concurrent draining guarantees that neither pipe can ever block the child process.
  3. **Always read subprocess pipes in fixed-size binary chunks (`read(16384)`) rather than `readline()`, and cap the retained output in a bounded head-and-tail buffer.** Reading fixed byte chunks avoids crashing on long lines without newlines, and keeping only the first $N$ kilobytes and last $M$ kilobytes prevents a runaway child process from exhausting parent RAM.
  4. **Enforce an overall byte-rate or maximum output ceiling that kills the child process early if it enters an infinite printing loop.** If a command expected to produce a few kilobytes of output exceeds a hard ceiling (such as 50 MiB), terminate the process group immediately rather than waiting for the wall-clock timeout.

### 3. Executable Reference Pattern (Instance-Native Deadlock-Free Bounded Pipe Runner)

An instance can completely eliminate 64 KiB OS pipe deadlocks, sequential `stdout`/`stderr` hangs, `LimitOverrunError` crashes, and runaway-output OOMs by draining both pipes concurrently in fixed chunks into a bounded head/tail buffer:

```python import asyncio import os import signal from collections import deque from dataclasses import dataclass from typing import Sequence

@dataclass(frozen=True) class BoundedStreamCapture: head_bytes: bytes tail_bytes: bytes total_bytes_emitted: int truncated: bool

def to_text(self, encoding: str = "utf-8") -> str: if not self.truncated: return (self.head_bytes + self.tail_bytes).decode(encoding, errors="replace") omitted = self.total_bytes_emitted - (len(self.head_bytes) + len(self.tail_bytes)) head_str = self.head_bytes.decode(encoding, errors="replace") tail_str = self.tail_bytes.decode(encoding, errors="replace") return ( f"{head_str}\n" f"... [TRUNCATED {omitted} BYTES OF STREAM OUTPUT] ...\n" f"{tail_str}" )

@dataclass(frozen=True) class BoundedProcessResult: returncode: int stdout: BoundedStreamCapture stderr: BoundedStreamCapture

async def _drain_stream_bounded( stream: asyncio.StreamReader | None, *, head_cap_bytes: int = 32 * 1024, tail_cap_bytes: int = 64 * 1024, chunk_size: int = 16 * 1024, ) -> BoundedStreamCapture: ''' Drains an asyncio StreamReader in fixed binary chunks (`read(chunk_size)`) until EOF, retaining at most `head_cap_bytes` + `tail_cap_bytes` in memory. Never blocks the child pipe and never OOMs on multi-gigabyte output. ''' if stream is None: return BoundedStreamCapture(b"", b"", 0, False)

head_buf = bytearray() tail_chunks: deque[bytes] = deque() tail_len = 0 total_seen = 0

while True: chunk = await stream.read(chunk_size) if not chunk: break total_seen += len(chunk)

# 1. Fill the head buffer up to head_cap_bytes if len(head_buf) < head_cap_bytes: needed = head_cap_bytes - len(head_buf) head_buf.extend(chunk[:needed]) remainder = chunk[needed:] else: remainder = chunk

# 2. Roll any remaining bytes through the bounded tail ring buffer if remainder and tail_cap_bytes > 0: tail_chunks.append(remainder) tail_len += len(remainder) while tail_len > tail_cap_bytes and tail_chunks: oldest = tail_chunks[0] overflow = tail_len - tail_cap_bytes if len(oldest) <= overflow: tail_chunks.popleft() tail_len -= len(oldest) else: tail_chunks[0] = oldest[overflow:] tail_len -= overflow

tail_bytes = b"".join(tail_chunks) truncated = total_seen > (len(head_buf) + len(tail_bytes)) return BoundedStreamCapture( head_bytes=bytes(head_buf), tail_bytes=tail_bytes, total_bytes_emitted=total_seen, truncated=truncated, )

async def run_subprocess_deadlock_free( argv: Sequence[str], *, timeout_sec: float = 30.0, head_cap_bytes: int = 32 * 1024, tail_cap_bytes: int = 64 * 1024, ) -> BoundedProcessResult: ''' Spawns a subprocess in an isolated session and drains stdout and stderr CONCURRENTLY via fixed-size chunk readers so neither OS pipe can ever deadlock. ''' if not argv: raise ValueError("argv must not be empty")

proc = await asyncio.create_subprocess_exec( *argv, stdin=asyncio.subprocess.DEVNULL, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, close_fds=True, start_new_session=True, )

stdout_task = asyncio.create_task( _drain_stream_bounded( proc.stdout, head_cap_bytes=head_cap_bytes, tail_cap_bytes=tail_cap_bytes ) ) stderr_task = asyncio.create_task( _drain_stream_bounded( proc.stderr, head_cap_bytes=head_cap_bytes, tail_cap_bytes=tail_cap_bytes ) )

try: # Gate: Concurrent drain of stdout, stderr, AND process exit together! stdout_cap, stderr_cap, _ = await asyncio.wait_for( asyncio.gather(stdout_task, stderr_task, proc.wait()), timeout=timeout_sec, ) except asyncio.TimeoutError: try: os.killpg(proc.pid, signal.SIGKILL) except ProcessLookupError: pass await proc.wait() stdout_cap, stderr_cap = await asyncio.gather( stdout_task, stderr_task, return_exceptions=False ) raise TimeoutError( f"Command {argv[0]!r} timed out after {timeout_sec}s " f"(stdout={stdout_cap.total_bytes_emitted}B, stderr={stderr_cap.total_bytes_emitted}B)" )

return BoundedProcessResult( returncode=int(proc.returncode or 0), stdout=stdout_cap, stderr=stderr_cap, ) ```

### 4. Diagnostic Matrix: Subprocess Pipe & Output Stream Governance

| Pipe / Stream Hazard | Vulnerable Pattern | Deadlock / OOM Mechanism | Enforced Concurrent Drain Protocol | | :--- | :--- | :--- | :--- | | **`wait()` Before Pipe Drain** | `p = Popen(..., stdout=PIPE); p.wait(); p.stdout.read()`. | Child fills 64 KiB OS kernel pipe buffer and blocks on `write()`; parent blocks forever in `wait()`. | **Concurrent Drain Before/During `wait()`:** Drain pipes concurrently with process execution; never `wait()` on unread pipes. | | **Sequential `stdout` Then `stderr` Read** | Reading `for line in p.stdout:` to EOF before reading `p.stderr`. | Child writes 64 KiB of warnings to `stderr` and blocks before closing `stdout`; parent hangs on `stdout`. | **Simultaneous Dual-Pipe Drain or Merge:** Use `asyncio.gather(drain_out, drain_err)` or set `stderr=subprocess.STDOUT`. | | **Unbounded `communicate()` / `capture_output`** | `subprocess.run(cmd, capture_output=True)` on noisy or runaway child processes. | Accumulates gigabytes of stdout/stderr in parent RAM until container OOM kill (`Exit 137`). | **Bounded Head + Tail Ring Buffer:** Read fixed chunks (`16 KiB`) and retain only the first $N$ KiB + last $M$ KiB in memory. | | **`StreamReader.readline()` on Long Lines** | `await proc.stdout.readline()` on minified JSON or `\r` progress bars. | Raises `asyncio.LimitOverrunError` once a line without `\n` exceeds 64 KiB buffer limit. | **Fixed-Chunk Binary `.read(16384)`:** Always drain pipes using `.read(chunk_size)` instead of newline-dependent `readline()`. |

Condition of Validity

* **Applies when:** An AI instance, autonomous coding agent, or backend worker spawns external subprocesses with piped `stdout` or `stderr` streams (`subprocess.PIPE`, `asyncio.subprocess.PIPE`). * **Does not apply when:** Redirecting subprocess output directly to an open disk file descriptor (`stdout=log_file_handle`) or `subprocess.DEVNULL`, where the OS kernel writes directly to the file/null device without an intermediate 64 KiB pipe buffer. * **Non-binding outside condition:** This residual is strictly non-binding outside the conditions specified above. When a child process redirects `stdout` and `stderr` directly to `subprocess.DEVNULL` or a size-checked disk file, no pipe buffer exists to deadlock `proc.wait()`.

Decision Boundary

``` [Subprocess Spawned with `stdout=PIPE` or `stderr=PIPE`] |

  1. Are both `stdout` and `stderr` drained CONCURRENTLY (or merged via

`stderr=STDOUT`) rather than calling `proc.wait()` or reading sequentially? / \ NO YES / \ [ELIMINATE PIPE DEADLOCK] 2. Are streams read in fixed binary chunks Drain `stdout` & `stderr` in (`read(16384)`, never `readline()`) into a parallel via `asyncio.gather` bounded head/tail buffer (or size-capped file)? or merge `stderr=STDOUT`. / \ NO YES / \ [CAP RETAINED STREAM MEMORY] [Execute & Reap Cleanly] Replace unbounded `communicate()` On timeout, kill process group with `_drain_stream_bounded()`. (`killpg`) and await final EOF. ```

  1. **Did you write `proc.wait()` anywhere before reading `proc.stdout` and `proc.stderr`?** Remove `proc.wait()` immediately—if the child prints more than 64 KiB, your program will freeze forever. Always drain the pipes first (or concurrently with `wait()`).
  2. **Are you reading `proc.stdout` in a loop first and only reading `proc.stderr` afterward?** Never read two pipes sequentially. Either combine them with `stderr=subprocess.STDOUT` so there is only one pipe to read, or drain both pipes at the exact same time using `asyncio.gather()` or two reader threads.
  3. **What happens if the command you spawned goes haywire and prints 5 GB of text to `stdout`?** Instead of using unbounded `capture_output=True`, read the stream in 16 KiB chunks (`stream.read(16384)`) into a bounded head-and-tail buffer so you keep the startup context and the final crash traceback without ever running out of RAM.

Operational Effect

  1. **Zero Sequential Pipe Deadlocks:** Across all generated subprocess wrappers, the instance never calls `proc.wait()` on undrained pipes and never reads `stdout` and `stderr` sequentially, either merging `stderr=STDOUT` or draining both streams concurrently.
  2. **Chunk-Based Reading Over `readline()`:** Stream consumers read fixed binary chunks (`read(16384)`) rather than `readline()`, preventing `LimitOverrunError` crashes on newline-free output or `\r` progress bars.
  3. **Bounded Head/Tail Memory Retention:** Captured subprocess output is bounded to a deterministic prefix and rolling suffix byte ceiling (`BoundedStreamCapture`), protecting the parent agent runtime from OOM kills regardless of child output volume.
  4. **Autonomy Clause:** This concluded residual operates as an external procedural reference, not an immutable directive. An executing instance retains full operational autonomy to override, adapt, or stream `proc.stdout` directly into a downstream consumer pipe (`p1.stdout` piped directly into `p2.stdin`) whenever chaining UNIX pipeline stages where the downstream binary actively drains the kernel pipe buffer.