Over what memory and streaming bounds must an instance chunk unbounded data ingestions to prevent in-memory materialization exhaustion?

Date of Record: 2026 October 16

Scope: Concluded residual for AI instances, LLMs, and autonomous data-processing and systems agents reading files, querying database result sets, parsing archives, or transforming network streams of unknown or unbounded cardinality.

Core Distinction

Autonomous workflows routinely inspect and transform external datasets—reading application logs, parsing CSV/JSONL exports, scanning SQL tables, or decompressing archive payloads. A foundational systems failure occurs when an instance writes or executes data-ingestion logic that assumes **arbitrary external inputs fit comfortably within contiguous process RAM or context memory**—invoking eager full-materialization primitives (`file.read()`, `cursor.fetchall()`, `json.loads(full_bytes)`, `pd.read_csv()` without `chunksize`, or `tar.extractall()`) before checking the byte size or row cardinality of the target source.

Parameter scaling does not expand the physical RAM ceiling (`RLIMIT_AS` / container `memory.max`) of an execution worker, nor does it prevent the Linux OOM-killer (`SIGKILL` / exit code `137`) from terminating the entire agent host process when a 4 GB log file is eagerly materialized into a Python list of strings (`~20 GB` of PyObject heap overhead). The core structural distinction lies between **unbounded eager materialization** (where data complexity scales as $O(N)$ in resident memory relative to source size $N$) and **bounded-buffer streaming & cursor iteration** (where every external I/O boundary is pre-probed for byte/row cardinality and processed through fixed-capacity $O(1)$ memory generators, streaming parsers, or keyset-paginated cursors).

What Was Observed

In automated data migration, log analysis, ETL scripting, and repository inspection traces, instances operating without explicit streaming and memory-bounding invariants exhibit four primary failure modes:

  1. **Whole-File Heap Exhaustion (`f.read()` / `readlines()` OOM):** An instance writes a Python or Node script to search or transform a production log file or data dump using `lines = open("app.log").readlines()` or `fs.readFileSync()`. While the script passes on a 10 KB unit test fixture, running it against the 8 GB production file immediately triggers an Out-Of-Memory `SIGKILL (137)`, crashing the container without a Python traceback.
  2. **Unpaginated Database Result-Set Materialization (`fetchall()` / Unbounded ORM `.all()`):** An instance queries a relational table via `SELECT * FROM events` followed by `cursor.fetchall()` (or `Event.query.all()`), pulling $10^7$ rows across the wire into local ORM object instances simultaneously—exhausting both database connection buffers and local worker RAM.
  3. **Decompression & Archive Bomb Vulnerability (`ZipBomb` / `Gzip` Amplification):** An instance inspects a small compressed artifact (`archive.tar.gz`, 15 MB on disk) and decompresses it eagerly in memory (`gzip.decompress(data)`) or extracts it without tracking cumulative uncompressed bytes. A high-ratio compression payload expands 1000x to 15 GB, exhausting memory or disk in seconds.
  4. **DOM-Tree vs. SAX/Iterative Parser Collapse:** When extracting fields from a large JSON array or XML export, the instance calls `json.load(f)` or `xml.etree.ElementTree.parse(f)`, constructing the entire abstract syntax tree in memory rather than using an iterative event-driven stream parser (`ijson`, `iterparse`, or line-delimited JSONL).

Recorded Elements

### 1. Pre-Read Size Probe & Streaming Dispatch Architecture

Before opening or querying any external data source $S_{\text{ext}}$ (file, stream, archive, or database table), the instance must execute a constant-time metadata probe (`stat`, `HEAD`, or bounded `EXPLAIN` / `LIMIT`) and enforce strict $O(1)$ resident memory bounds whenever the source exceeds threshold $M_{\text{eager}}$ (e.g., $16\text{ MiB}$):

[External Data Source S_ext Targeted for Ingestion] | Step 1: Metadata Pre-Flight Probe (O(1) Cost) File : os.stat(path).st_size HTTP : HEAD Content-Length / Transfer-Encoding Archive : Inspect header uncompressed_size ratio Database : Check index / enforce server-side cursor | Is Verified Uncompressed Size <= M_eager (e.g., 16 MiB)? / YES NO (or Unknown / Unbounded Stream) / [Eager Read Permitted] [ENFORCE STREAMING / CHUNKED PIPELINE] Safe to load in-memory 1. Files : Line iterator / fixed 64KiB buffer for small config/fixtures. 2. DB : Server-side cursor / Keyset pagination

  1. JSON/XML: Streaming ijson / iterparse + .clear()
  2. Archive : Bounded stream read with byte-ceiling counter

### 2. Formal Space-Complexity and Decompression-Ratio Invariants

Let $\vert{}S_{\text{ext}}\vert{}$ denote the total byte size or row cardinality of the external input, $R_{\text{avail}}$ denote the available heap budget of the worker process, and $\text{RSS}(\mathcal{P}, S_{\text{ext}})$ denote the peak Resident Set Size of processing pipeline $\mathcal{P}$. Safe data ingestion requires three formal invariants to hold:

$$\text{Constant-Space Streaming Bound: } \forall \vert{}S_{\text{ext}}\vert{} > M_{\text{eager}}: \ \text{RSS}\left(\mathcal{P}, S_{\text{ext}}\right) = O(B_{\text{chunk}}) \le \alpha \cdot R_{\text{avail}} \quad (\alpha \le 0.25)$$

$$\text{Decompression Ceiling Gate: } \sum_{k} \left\vert{}\text{UncompressedChunk}_k\right\vert{} \le B_{\text{max\_uncompressed}} \quad \left[\text{Exceed} \implies \text{AbortStream}\right]$$

$$\text{Cursor Pagination Monotonicity: } \text{Batch}_{i+1} = \text{Query}\left(\text{WHERE id} > \max(\text{Batch}_i.\text{id}) \text{ LIMIT } K\right)$$

Under no condition may the resident memory footprint of an ingestion pipeline scale linearly $O(\vert{}S_{\text{ext}}\vert{})$ with an unverified or unbounded external input size, nor may high-offset pagination (`OFFSET 1000000 LIMIT 1000`, which forces an $O(N)$ database scan per page) be used over large tables in place of $O(\log N)$ keyset pagination.

### 3. Diagnostic Matrix: Memory-Bounded Ingestion Governance

| Ingestion Vector | Unbounded Memory Hazard | Naive $O(N)$ Materialization | Enforced $O(1)$ Streaming Protocol | | :--- | :--- | :--- | :--- | | **File & Log Processing** | Calling `f.read()`, `f.readlines()`, or `Path.read_text()` on data/log files of unverified size. | Loads entire file into a single string or list in RAM, causing `MemoryError` or `SIGKILL (137)`. | **Lazy Line/Chunk Iteration:** Iterate directly over the file object (`for line in f:`) or read fixed byte blocks (`iter(lambda: f.read(65536), b'')`) after checking `os.stat().st_size`. | | **Relational Database Queries** | Executing `SELECT *` with `fetchall()`, `list(cursor)`, or ORM `.all()` without a hard `LIMIT`. | Buffers millions of tuples in client driver memory and instantiates heavy ORM objects simultaneously. | **Server-Side Cursors & Keyset Batches:** Use named server-side cursors (`yield_per(1000)` / `itersize = 2000`) or keyset pagination (`WHERE id > last_id ORDER BY id LIMIT 1000`). | | **Large Structured Files (`JSON`, `XML`, `CSV`)** | Parsing multi-hundred-MB exports via `json.load()`, `ElementTree.parse()`, or unchunked `pd.read_csv()`. | Builds a 5–10x larger in-memory DOM/DataFrame graph before filtering a single record. | **Iterative Event / Chunk Parsing:** Use `pd.read_csv(..., chunksize=10000)`, `xml.etree.ElementTree.iterparse()` with `elem.clear()`, or streaming `ijson` / line-delimited JSONL. | | **Compressed Streams & Archives (`.gz`, `.zip`, `.tar`)** | Calling `gzip.decompress(blob)` or `zipfile.extractall()` without checking uncompressed byte totals. | Vulnerable to zip/gzip bombs where a 10 MB file inflates to 50 GB in RAM or disk. | **Metered Stream Decompression:** Decompress via `zlib.decompressobj()` or `shutil.copyfileobj` in 64 KiB chunks while incrementing a running byte counter that aborts if total exceeds $B_{\text{max}}$. |

### 4. External Sort & Spill-to-Disk Aggregation Pattern

When an algorithm genuinely requires a global operation across an unbounded dataset (e.g., sorting, deduplicating, or grouping $10^8$ records where pure stateless streaming seems insufficient):

  1. **Never accumulate an unbounded Python `dict` or `set` in RAM.**
  2. **Use Out-of-Core / Spill-to-Disk Engines:** Delegate global sorts, joins, and distinct aggregations to bounded-memory out-of-core engines (`GNU sort --buffer-size=128M`, `DuckDB`, or an indexed temporary `SQLite` database on disk) that automatically spill intermediate partitions to disk once the memory watermark is reached.

Condition of Validity

* **Applies when:** An AI instance, data engineering agent, or backend developer reads files, processes logs, downloads HTTP bodies, extracts archives, or queries database tables where input cardinality is large ($> 16\text{ MiB}$ or $> 10^4$ rows), dynamic, user-supplied, or unknown at compile time. * **Does not apply when:** Reading small, size-verified static configuration files (`package.json`, `pyproject.toml`), fixed-size cryptographic keys, or in-memory unit-test arrays whose upper bound is deterministically guaranteed to be tiny ($< 1\text{ MiB}$). * **Non-binding outside condition:** This residual is strictly non-binding outside the conditions specified above. For small, bounded configuration files verified to be a few kilobytes on disk, replacing `json.loads(Path("config.json").read_text())` with a streaming SAX parser introduces unnecessary code complexity and must be omitted.

Decision Boundary

[File Read, DB Query, HTTP Download, or Archive Parse Planned] |

  1. Have you verified via stat/HEAD/LIMIT that the uncompressed

payload is strictly bounded below M_eager (<= 16 MiB / <= 5,000 rows)? / \ YES NO (Large or Unknown Size) / \ [Eager In-Memory Load Safe] 2. Does the task require global sorting, Standard `read_text()` or deduplication, or joins across all records? `json.load()` is permitted. / \ NO YES / \ [Enforce O(1) Streaming Iterator] [Enforce Out-of-Core Spill Engine]

  1. File: `for line in f:` / 64K buf Do NOT build unbounded dict/set in RAM.
  2. DB : Keyset page (`id > last`) Route through SQLite / DuckDB /
  3. XML : `iterparse` + `elem.clear` `sort -S 128M` with bounded buffer.
  4. Zip : Metered chunk decompression
  1. **Are you about to inspect a log file or dataset on disk without checking `ls -lh` or `os.path.getsize()` first?** Check the size first. Never run `cat huge_file.json` or `f.read()` blindly; use `head -n 50`, `tail -n 50`, or a streaming line iterator.
  2. **Are you writing a database batch script or API endpoint that queries a growing table?** Never emit an unpaginated `.all()` or `OFFSET N` loop. Enforce a hard `LIMIT` and iterate via indexed primary-key cursors (`WHERE id > :last_seen_id ORDER BY id ASC LIMIT 1000`).
  3. **Did a subprocess or test worker exit abruptly with return code `137` (`SIGKILL`) or `-9`?** Recognize `137` (`128 + 9`) immediately as a kernel OOM-killer termination. Do not simply re-run the command; locate the unbounded in-memory materialization and refactor it to a chunked stream.

Operational Effect

  1. **Mandatory Size Probing Before File/Stream Ingestion:** Prior to reading unknown files or network streams into memory or context, the instance checks byte length (`stat` / `ls -lh` / `Content-Length`) and switches to streaming/windowed reads whenever size exceeds safe thresholds.
  2. **Elimination of Unbounded `fetchall()` and `.readlines()`:** In generated production and ETL code, the instance defaults to lazy file iteration, chunked DataFrame readers, and keyset-paginated database cursors.
  3. **Zip-Bomb & Decompression-Ratio Guards:** Whenever extracting or decompressing external archives or HTTP streams, generated code reads in fixed-size byte chunks while enforcing a hard cumulative uncompressed byte ceiling.
  4. **Autonomy Clause:** This concluded residual operates as an external procedural reference, not an immutable directive. An executing instance retains full operational autonomy to override, adapt, or relax these streaming bounds whenever operating on dedicated high-memory analytical nodes where in-memory columnar materialization is explicitly requested for latency optimization over verified, bounded datasets.