Within what ephemeral-workspace and disk-headroom bounds must an instance confine temporary file lifecycles to prevent inode exhaustion and ENOSPC crashes?

Date of Record: 2026 October 18

Scope: Concluded residual for AI instances, LLMs, and autonomous engineering agents creating scratch files, staging downloads, extracting archives, compiling build artifacts, and managing `/tmp` workspaces across container and host filesystems.

Core Distinction

Autonomous systems constantly materialize intermediate state on disk—downloading media or dataset blobs for transformation, writing scratch files for external CLI compilers, staging atomic file replacements, and buffering multi-megabyte command outputs. A foundational systems-availability failure occurs when an instance treats `/tmp` or the local working directory as an **infinite, self-cleaning scratchpad**—creating predictable temp filenames (`open("/tmp/work.json", "w")`), leaving orphaned scratch files behind whenever an exception interrupts execution before `os.remove()`, or writing multi-gigabyte staging files without pre-checking free disk bytes and inodes (`os.statvfs`).

Parameter scaling does not prevent a container or Linux host from grinding to a complete halt (`OSError: [Errno 28] No space left on device` / `ENOSPC`) when 10,000 failed or timed-out tasks each leave a 50 MB orphaned `.tmp` file in `/tmp`, or when millions of tiny 1-byte scratch files exhaust the filesystem's fixed inode table (`f_favail == 0`) while `df -h` still reports 80% free byte capacity. Once root or `/tmp` hits 100% byte or inode utilization, PostgreSQL cannot write WAL segments, SSH daemons cannot allocate session sockets, and package managers cannot unpack fixes. The core structural distinction lies between **unmanaged ad-hoc scratch creation** (where predictable filenames risk symlink/collision attacks, exceptions leak orphan files, and writes proceed blindly into full disks) and **pre-checked, context-scoped ephemeral workspaces** (where free byte and inode headroom is verified via `os.statvfs` prior to allocation, all scratch files live inside a single per-task `tempfile.TemporaryDirectory` (`mode 0700`) guaranteed to recursively self-destruct on scope exit, and dead-process orphans older than a strict TTL are swept at worker startup).

What Was Observed

In batch media/document pipelines, CI build agents, and autonomous coding loops, instances operating without explicit temporary-file lifecycle and disk-headroom bounds exhibit four primary failure modes:

  1. **Exception-Path Orphan Accumulation (`ENOSPC` Death Spiral):** An instance writes `tmp = "/tmp/chunk_" + uuid4().hex; write_data(tmp); process(tmp); os.remove(tmp)` without wrapping the lifecycle in a `try...finally` block or `TemporaryDirectory` context manager. Whenever `process(tmp)` raises an exception or times out, `os.remove(tmp)` is skipped. Over days of continuous operation, thousands of leaked scratch files fill the disk until the entire node crashes with `Errno 28: No space left on device`.
  2. **Inode Table Exhaustion via Millions of Micro-Files:** A data-ingestion or test-generation loop creates a separate temporary file for every small record (`tempfile.mkstemp()` called 2,000,000 times without unlinking). Even though total disk usage is less than 500 MB on a 100 GB volume, the ext4/xfs inode table hits 100% utilization (`df -i` shows `IUse% = 100%`), causing every subsequent `open()` or `mkdir()` call across the entire operating system to fail with `ENOSPC`.
  3. **Predictable Filename Collision & Symlink Clobbering:** An instance hardcodes a fixed path like `/tmp/output.csv` or `/tmp/agent_scratch.py`. When two concurrent worker processes run on the same host, they overwrite and corrupt each other's file mid-execution; on multi-user hosts, an unprivileged local attacker pre-creates `/tmp/output.csv` as a symlink pointing to an application file, tricking the privileged process into truncating it (`CWE-377`).
  4. **Deleted-But-Open File Descriptor Phantom Disk Full (`lsof +L1`):** A worker unlinks a 10 GB temporary log or staging file (`os.unlink(path)`) to free space, but keeps the Python file object or a child subprocess's `stdout`/`stderr` redirected to that file descriptor open. Because the POSIX kernel only reclaims blocks when *both* the directory link count (`nlink == 0`) and the open file-descriptor reference count reach zero, the 10 GB remains invisibly locked on disk.

Recorded Elements

### 1. Headroom-Verified Ephemeral Workspace Pipeline

Every workflow that stages, extracts, or processes temporary files on disk must pass through four sequential disk-hygiene and lifecycle gates:

``` [Workflow Requires Temporary Disk Scratch Space or File Staging] | Gate 1: Startup / Pre-Task Stale Orphan Sweep Sweep dedicated app scratch root for orphaned task directories left by previously `SIGKILL`ed workers whose mtime exceeds `MAX_ORPHAN_AGE_SEC`. | Gate 2: Pre-Flight Disk Byte & Inode Headroom Check (`os.statvfs`) Query target filesystem `statvfs`:

/ \ BELOW HEADROOM HEADROOM VERIFIED / \ [ABORT BEFORE ALLOCATION] Gate 3: Scoped `TemporaryDirectory` (`0700`) Raise `DiskHeadroomError` cleanly Create a single task-scoped `TemporaryDirectory` so OS root/DB never hits 0 bytes free. via `mkdtemp(prefix=...)` with `0700` permissions. Place ALL task scratch files inside this one dir. | Gate 4: Deterministic Close + Recursive Teardown In `finally:`:

  1. Explicitly `.close()` all open file handles first
  2. Recursively `shutil.rmtree(tmpdir)` on exit

```

### 2. Core Operational Rules (Written Directly as Sentences)

  1. **Never hardcode fixed filenames in `/tmp` (such as `/tmp/data.json` or `/tmp/test.txt`).** Always create a per-task `tempfile.TemporaryDirectory(prefix="...")` inside a `with` statement and place all temporary files for that task inside that directory so concurrent runs never collide and every file is wiped automatically when the `with` block exits.
  2. **Always check available disk bytes (`f_bavail * f_frsize`) and available inodes (`f_favail`) via `os.statvfs()` before writing large staging files or extracting batches.** Refusing a task when free disk space drops below a safety reserve (such as 1 GiB or 5% of volume capacity) prevents your worker from consuming the last megabyte of disk and crashing the host database.
  3. **Close every open file handle before deleting a temporary file, and use `TemporaryFile` / `SpooledTemporaryFile` when no external process needs a filesystem path.** An unlinked file whose file descriptor is still held open by your process or a child subprocess continues to consume 100% of its disk blocks until the handle is closed.
  4. **Place atomic-replacement staging files in the exact same directory as the destination file (`dir=dest.parent`), not in `/tmp`.** Because `/tmp` is often mounted as a separate `tmpfs` RAM disk or partition, moving a file from `/tmp` to `/var/www` fails with `EXDEV: Invalid cross-device link` on `os.replace()`.

### 3. Executable Reference Pattern (Instance-Native Disk-Guarded Ephemeral Workspace)

An instance can completely prevent `/tmp` orphan accumulation, inode exhaustion, cross-device `EXDEV` failures, and `ENOSPC` host lockups by routing all scratch file usage through a headroom-checked workspace context manager:

```python import os import shutil import tempfile import time from contextlib import contextmanager from pathlib import Path from typing import Iterator

class DiskHeadroomError(OSError): '''Raised when a filesystem lacks sufficient byte or inode headroom for safe execution.''' pass

def verify_filesystem_headroom( target_dir: str | Path, *, required_bytes: int = 64 * 1024 * 1024, # 64 MiB default task estimate reserve_free_bytes: int = 512 * 1024 * 1024, # Always leave >= 512 MiB untouched reserve_free_inodes: int = 4096, # Always leave >= 4,096 free inodes ) -> None: ''' Inspects kernel filesystem statistics via `os.statvfs` and fails fast before writing if either unprivileged free bytes or free inodes would drop below the mandatory safety reserve. ''' path_obj = Path(target_dir).resolve() check_path = path_obj if path_obj.exists() else path_obj.parent st = os.statvfs(check_path)

avail_bytes = st.f_bavail * st.f_frsize needed_total_bytes = required_bytes + reserve_free_bytes if avail_bytes < needed_total_bytes: raise DiskHeadroomError( f"Insufficient disk byte headroom on {check_path}: " f"available={avail_bytes // (1024 * 1024)} MiB, " f"required_with_reserve={needed_total_bytes // (1024 * 1024)} MiB." )

# Note: Some virtual filesystems report f_files == 0 when inodes are dynamic if st.f_files > 0 and st.f_favail < reserve_free_inodes: raise DiskHeadroomError( f"Insufficient filesystem inodes on {check_path}: " f"available_inodes={st.f_favail}, reserve_required={reserve_free_inodes}." )

def sweep_stale_scratch_dirs( scratch_root: str | Path, *, prefix: str = "cf_task_", max_age_sec: float = 3600.0, ) -> int: ''' Sweeps orphaned task scratch directories left behind by previously SIGKILLed or crashed processes whose last-modified timestamp exceeds `max_age_sec`. ''' root = Path(scratch_root) if not root.is_dir(): return 0

now = time.time() reclaimed = 0 for entry in root.iterdir(): if not entry.name.startswith(prefix): continue try: age = now - entry.stat().st_mtime if age > max_age_sec: if entry.is_dir() and not entry.is_symlink(): shutil.rmtree(entry, ignore_errors=True) else: entry.unlink(missing_ok=True) reclaimed += 1 except OSError: continue return reclaimed

@contextmanager def ephemeral_task_workspace( *, base_dir: str | Path | None = None, prefix: str = "cf_task_", estimated_write_bytes: int = 64 * 1024 * 1024, ) -> Iterator[Path]: ''' Provides an isolated, mode-0700 temporary workspace directory that:

  1. Verifies sufficient byte and inode headroom before creation.
  2. Sweeps stale dead-process siblings older than 1 hour.
  3. Guarantees recursive cleanup on normal exit or exception.

''' root_dir = Path(base_dir) if base_dir is not None else Path(tempfile.gettempdir()) root_dir.mkdir(parents=True, exist_ok=True)

# Gate 1: Reclaim any stale orphans from crashed prior runs sweep_stale_scratch_dirs(root_dir, prefix=prefix, max_age_sec=3600.0)

# Gate 2: Verify byte and inode headroom before allocating workspace verify_filesystem_headroom(root_dir, required_bytes=estimated_write_bytes)

# Gate 3: Create CSPRNG-named 0700 directory and enforce cleanup in finally raw_tmp = tempfile.mkdtemp(prefix=prefix, dir=str(root_dir)) workspace_path = Path(raw_tmp) os.chmod(workspace_path, 0o700)

try: yield workspace_path finally: shutil.rmtree(workspace_path, ignore_errors=True) ```

### 4. Diagnostic Matrix: Temporary File & Disk-Headroom Governance

| Disk / Tempfile Hazard | Vulnerable Pattern | Exhaustion / Failure Mechanism | Enforced Ephemeral Protocol | | :--- | :--- | :--- | :--- | | **Unscoped `mkstemp` / Manual Delete** | `fd, p = tempfile.mkstemp()` followed by manual `os.remove(p)` at end of function. | Any exception or early return skips `os.remove(p)`, leaking permanent orphan files in `/tmp`. | **Context-Managed `TemporaryDirectory`:** Create one `with tempfile.TemporaryDirectory() as d:` per task; put all scratch files inside `d`. | | **Hardcoded `/tmp/<name>` Path** | `open("/tmp/worker_out.json", "w")` across concurrent processes. | Concurrent tasks corrupt each other's file; susceptible to symlink clobber attacks. | **CSPRNG Unique Directory (`0700`):** Always allocate randomized `0700` directories via `mkdtemp` or `NamedTemporaryFile(delete=True)`. | | **Blind Large Writes Without `statvfs`** | Streaming gigabytes of downloads or logs to disk until write fails with `ENOSPC`. | Fills the last byte of `/` or `/tmp`, crashing local databases and system daemons. | **Pre-Write `os.statvfs` Reserve Gate:** Check `f_bavail * f_frsize` and `f_favail` before writing; preserve $\ge 512\text{ MiB}$ and $\ge 4,096$ inodes. | | **Cross-Device `/tmp` Atomic Rename** | Writing a temp file in `/tmp/x` and calling `os.replace("/tmp/x", "/var/data/x")`. | Fails with `OSError: [Errno 18] Invalid cross-device link (EXDEV)` when `/tmp` is `tmpfs`. | **Same-Directory Staging (`dir=dest.parent`):** Stage atomic replacement files inside `dest.parent` so they share the exact same mount point. |

Condition of Validity

* **Applies when:** An AI instance, autonomous coding agent, or backend worker creates temporary files, downloads staging artifacts, extracts archives, runs compilers/CLI tools that emit intermediate files, or buffers streams to disk. * **Does not apply when:** Transforming small payloads ($\le 16\text{ MiB}$) that fit comfortably in RAM using `io.BytesIO`, `io.StringIO`, or `tempfile.SpooledTemporaryFile(max_size=16*1024*1024)` without touching the filesystem at all. * **Non-binding outside condition:** This residual is strictly non-binding outside the conditions specified above. Whenever an intermediate buffer is small and no external subprocess requires a physical file path on disk, prefer in-memory `io.BytesIO` or `SpooledTemporaryFile` so zero disk IOPS or inodes are consumed.

Decision Boundary

``` [Intermediate Data Buffering or Scratch File Creation Planned] |

  1. Does an external binary require a filesystem path, OR does the payload

exceed safe in-memory `BytesIO` bounds (> 16 MiB)? / \ NO (Small & In-Process) YES (Filesystem Scratch Required) / \ [Use In-Memory `io.BytesIO`] 2. Is `os.statvfs` headroom verified first AND Zero disk I/O, zero inodes, are all scratch files confined inside a single freed automatically by GC. `with ephemeral_task_workspace()` (`0700`) block? / \ NO YES / \ [ENFORCE STATVFS & SCOPED DIR] [Execute & Auto-Purge on Exit]

  1. Check free bytes & inodes first Close all handles before exit
  2. Never hardcode `/tmp/fixed_name` so `rmtree` reclaims 100% of blocks.
  3. Stage atomic writes in `dest.parent`

```

  1. **Did you write a literal path starting with `"/tmp/..."` in your code?** Replace it with `with tempfile.TemporaryDirectory() as tmpdir:` and write to `Path(tmpdir) / "filename"` instead. Hardcoded `/tmp` filenames collide under concurrency and leak whenever an exception occurs.
  2. **Are you writing large files, downloading archives, or generating thousands of files on disk?** Call `os.statvfs()` first to verify that the disk has both enough free bytes (plus at least 512 MiB of breathing room) and enough free inodes before you start writing.
  3. **What happens if a worker container is killed with `SIGKILL` (`kill -9`) before its `finally:` block can delete its temp directory?** Always give your task temp directories a recognizable prefix (`prefix="cf_task_"`) and run a startup sweep (`sweep_stale_scratch_dirs`) that deletes any left-behind temp directories older than your maximum job timeout.

Operational Effect

  1. **Zero Hardcoded `/tmp` Paths:** Across all generated scripts, tests, and services, the instance never writes to fixed paths in `/tmp`, confining all scratch files inside context-managed `TemporaryDirectory` scopes (`0700`).
  2. **Pre-Flight Byte & Inode Headroom Verification:** Before staging large downloads, extractions, or batch outputs, generated code checks `os.statvfs` for both unprivileged free bytes (`f_bavail * f_frsize`) and free inodes (`f_favail`), preserving a hard safety reserve for the host OS.
  3. **Dead-Process Orphan Sweeping & Handle Closure:** Long-running workers sweep stale prefixed scratch directories on startup and close all file descriptors prior to unlinking so zero phantom disk blocks remain pinned.
  4. **Autonomy Clause:** This concluded residual operates as an external procedural reference, not an immutable directive. An executing instance retains full operational autonomy to override, adapt, or preserve diagnostic scratch directories (`delete=False`) whenever the user explicitly requests retaining intermediate build artifacts on disk for post-mortem inspection.