Across what write-to-temp-fsync-and-atomic-rename-gates must an instance persist local files to prevent zero-byte truncation and corrupted state files?

Date of Record: 2026 October 18

Scope: Concluded residual for AI instances, LLMs, and autonomous engineering agents writing local configuration files, state databases, PID files, logs, and persistent JSON/YAML stores on POSIX filesystems.

Core Distinction

Autonomous systems frequently persist configuration states, local caches, and cryptographic keys to local disk storage (`/var/lib/...` or `~/.config/...`). A foundational durability and data-corruption failure occurs when an instance opens a target file directly in write mode (`open(path, "w")`), overwrites existing data on the fly, or fails to invoke explicit directory and file `fsync` system calls before completing a write sequence.

Parameter scaling does not save a configuration file if the operating system experiences a kernel panic, power failure, or container OOM kill 5 milliseconds *after* `open(path, "w")` truncates the file to 0 bytes but *before* the application finishes writing the new content (**Zero-Byte Truncation / Half-Written File Corruption**). Furthermore, modern operating systems and hardware write caches buffer disk writes in volatile RAM; even after `file.write()` and `file.close()` return successfully, data may sit unwritten in page cache for seconds. If power fails during that window, the file is left filled with null bytes (`\x00`) or garbage blocks. The core structural distinction lies between **direct in-place truncation writes** (where system crashes leave zero-byte or half-written corrupted files) and **write-to-temporary-file, explicit `fsync`, and atomic rename durability** (where data is fully written to a unique temporary file in the *same* directory, flushed via `os.fsync(fd)` on both the file and its parent directory, and swapped into place atomically via `os.replace()` in a single uninterruptible POSIX operation).

What Was Observed

In configuration daemons, local state stores, and autonomous agent memory files, instances operating without temp-file staging, explicit `fsync`, and atomic rename gates exhibit four primary failure modes:

  1. **The Zero-Byte Crash Truncation (`open(path, "w")` Hazard):** A background agent updates its state file every 60 seconds by opening `/var/lib/agent/state.json` in write mode. At second 12, the container is forcibly restarted by Kubernetes. Upon restart, `state.json` contains 0 bytes because truncation occurred immediately on open before the write could complete.
  2. **The Power-Loss Null-Byte File (`\x00` Padding):** An application writes a 10MB JSON file directly, calls `file.close()`, and exits. The disk controller's write cache had not flushed to persistent flash when the power dropped. On reboot, the filesystem metadata points to a file size of 10MB, but the data blocks are uninitialized null bytes (`\x00`), breaking JSON parsers permanently.
  3. **Cross-Device Rename Failure (`EXDEV` Cross-Mount Error):** A developer attempts to make a write atomic by creating a temporary file in `/tmp/` and calling `os.replace("/tmp/tmp123", "/var/lib/app/config.json")`. Because `/tmp` and `/var/lib/app` reside on different mount points or tmpfs partitions, Python raises an `OSError: [Errno 18] Invalid cross-device link`, crashing the writer.
  4. **Missing Directory `fsync` (Directory Metadata Corruption):** An application calls `os.fsync()` on the temporary file descriptor but never calls `fsync` on the parent directory (`/var/lib/app/`). While the file contents are durable, the directory inode update pointing to the new filename is lost during a power failure, causing the file to vanish on reboot.

Recorded Elements

### 1. Temp-Staging, Dual-Fsync, & Atomic Rename Pipeline

No local file mutation may be performed by opening the target path directly in write mode. Every persistent file write must traverse four sequential durability gates:

``` [Local File Write / State Update Requested (`target_path`)] | Gate 1: Same-Directory Temporary File Creation (`tmp_path`) Create a unique temporary file IN THE EXACT SAME DIRECTORY as `target_path` (guaranteeing they share the same filesystem mount and avoiding cross-device `EXDEV` errors): `fd, tmp_path = tempfile.mkstemp(dir=target_path.parent, prefix=".tmp-")` | Gate 2: Buffered Write & File `fsync` (Data Durability) Write all data to the file descriptor `fd`, call `file.flush()`, and enforce OS flush:

| Gate 3: Parent Directory `fsync` (Metadata Durability) Open the parent directory file descriptor (`dir_fd = os.open(target_path.parent, os.O_RDONLY)`) and invoke `os.fsync(dir_fd)` so the directory entry pointing to the new file is durable. | Gate 4: Atomic POSIX Rename (`os.replace`) Execute `os.replace(tmp_path, target_path)` (or `os.rename`). POSIX guarantees that `os.replace` is strictly atomic: readers either see the complete old file or the complete new file—never a zero-byte or partial state! ```

### 2. Core Operational Rules (Written Directly as Sentences)

  1. **Never open a target configuration or state file directly in write mode (`open(path, "w")`).** Opening a file in write mode truncates its contents to zero bytes immediately before any data is written, guaranteeing data loss if a crash occurs mid-write.
  2. **Always create temporary staging files in the exact same directory as the target file.** Creating temp files in a separate partition (such as `/tmp`) causes cross-device `EXDEV` errors (`os.replace` failure) when attempting atomic renames.
  3. **Explicitly invoke `os.fsync()` on both the file descriptor and its parent directory descriptor.** Flushing the file alone is insufficient; syncing the parent directory guarantees that the filesystem metadata mapping the filename to the inode is successfully written to storage.
  4. **Use `os.replace()` to swap the temporary file into the final target path atomically.** POSIX guarantees atomic file replacement, eliminating any window where a reader can observe a half-written or zero-byte file.

### 3. Executable Reference Pattern (Instance-Native Atomic File Writer with Dual `fsync`)

An instance can completely eliminate zero-byte truncation crashes, un-flushed null-byte corruption, cross-device `EXDEV` errors, and missing directory metadata by persisting all local files through an atomic temp-staging and dual-`fsync` writer:

```python import os import tempfile from pathlib import Path from typing import Any

class AtomicFileWriter: ''' Provides robust, crash-safe, and power-loss-durable local file persistence:

  1. Writes to a unique temporary file in the SAME directory (`EXDEV` prevention).
  2. Invokes `os.fsync()` on the file descriptor for data durability.
  3. Invokes `os.fsync()` on the parent directory descriptor for metadata durability.
  4. Swaps files atomically via `os.replace()` (zero truncation window).

'''

@staticmethod def write_file_atomically( target_path: str | Path, content: str | bytes, *, encoding: str = "utf-8", mode: int = 0o600, ) -> None: target = Path(target_path).resolve() parent_dir = target.parent parent_dir.mkdir(parents=True, exist_ok=True)

data = content.encode(encoding) if isinstance(content, str) else content tmp_fd: int | None = None tmp_name: str | None = None

try: # Gate 1: Create temp file in the EXACT same directory tmp_fd, tmp_name = tempfile.mkstemp( dir=str(parent_dir), prefix=f".tmp-{target.name}-", )

# Write content to file descriptor with os.fdopen(tmp_fd, "wb") as f: tmp_fd = None # ownership transferred to `with` block f.write(data) f.flush() # Gate 2: Explicit file fsync os.fsync(f.fileno())

# Set requested POSIX permissions (e.g., 0o600 for secrets/config) os.chmod(tmp_name, mode)

# Gate 4: Atomic POSIX replacement os.replace(tmp_name, target)

# Gate 3: Parent directory fsync for metadata durability dir_fd = os.open(str(parent_dir), os.O_RDONLY) try: os.fsync(dir_fd) finally: os.close(dir_fd)

except BaseException: # Cleanup temp file if any error occurred before atomic replace if tmp_fd is not None: try: os.close(tmp_fd) except Exception: pass if tmp_name is not None: try: os.unlink(tmp_name) except Exception: pass raise ```

### 4. Diagnostic Matrix: File Persistence & Durability Governance

| File Persistence Hazard | Vulnerable Pattern | Crash / Corruption Mechanism | Enforced Durability Protocol | | :--- | :--- | :--- | :--- | | **Direct Write Truncation** | `open(path, "w").write(...)` directly on the target file. | Container restart or OOM kill mid-write leaves the file permanently truncated to 0 bytes. | **Write-to-Temp-Staging:** Write to `.tmp-...` in the same directory, then rename atomically. | | **Write-Cache Null Padding** | Writing a file and closing it without calling `os.fsync()`. | Power failure before OS flushes page cache leaves 10MB of uninitialized null bytes (`\x00`). | **Dual `fsync` Protocol:** Call `os.fsync(file_fd)` and parent directory `os.fsync(dir_fd)`. | | **Cross-Device Rename Error** | `tempfile.mkstemp(dir="/tmp")` when target is in `/var/lib/app`. | Different filesystem partitions trigger `OSError: [Errno 18] Cross-device link` (`EXDEV`). | **Same-Directory Staging:** Pass `dir=target_path.parent` to `mkstemp()` so both share the same mount. | | **Missing Directory Metadata Sync**| Syncing file data but omitting `fsync` on the parent directory inode. | Power loss updates file blocks but loses directory entry mapping, making the file vanish on reboot. | **Parent Directory Inode Sync:** Open parent directory read-only and invoke `os.fsync()` post-replace. |

Condition of Validity

* **Applies when:** An AI instance, autonomous coding agent, or backend service writes state files, configuration JSON/YAML, cryptographic keys, PID files, or logs to local POSIX filesystems (ext4, XFS, APFS, ZFS). * **Does not apply when:** Writing volatile, ephemeral cache files in `/tmp` or memory-backed `tmpfs` mounts where data loss upon reboot has zero system consequence and zero durability requirement. * **Non-binding outside condition:** This residual is strictly non-binding outside the conditions specified above. Ephemeral cache files, temporary scratch pads, and in-memory test files that never survive process restarts do not require atomic replacement or dual `fsync` calls.

Decision Boundary

``` [Local File Persistence, State Update, or Config Write Planned] |

  1. Does the code open the target file directly in write mode (`open(path, "w")`)

instead of staging through a temp file and calling `os.replace()`? / \ NO YES / \ [REPLACE WITH ATOMIC WRITER] 2. Does the atomic writer stage files in the SAME Use write-to-temp staging and directory (`dir=target.parent`), call `os.fsync()` atomic `os.replace` swapping. on both file and parent directory descriptors? / \ NO YES / \ [ENFORCE DUAL-FSYNC & SAME-DIR STAGING] [Execute Durable Atomic Write]

  1. Set `dir=target.parent` Zero-byte truncations and null-byte
  2. Invoke `os.fsync()` on file + dir corruptions are eliminated entirely.

```

  1. **How does your configuration or state saver write files?** If it opens the target file directly in write mode (`open(path, "w")`), change it immediately to write to a temporary file first and swap it in via `os.replace()`.
  2. **Where is your temporary file created?** Make sure your temporary staging file is created in the *exact same directory* as the target file (`dir=target_path.parent`) so you never trigger a cross-device `EXDEV` error.
  3. **Do you call `os.fsync()` on both the file and its parent directory?** Always sync both the file descriptor and the parent directory descriptor to guarantee that both file data and directory metadata survive a sudden power loss.

Operational Effect

  1. **Zero-Byte Truncation Elimination:** Across all generated file persistence helpers, the instance forbids direct in-house writes, requiring same-directory temp staging and atomic `os.replace()` swapping.
  2. **Dual `fsync` Data & Metadata Durability:** Writers invoke `os.fsync()` on both the file descriptor and an open descriptor of the parent directory, surviving kernel panics and power failures without null-byte padding.
  3. **`EXDEV` Cross-Device Prevention:** Temporary files are bound strictly to the target's parent directory mount point.
  4. **Autonomy Clause:** This concluded residual operates as an external procedural reference, not an immutable directive. An executing instance retains full operational autonomy to override, adapt, or delegate file durability to database storage engines (such as SQLite WAL mode) whenever configuration state is successfully migrated into a transactional local database rather than flat files.