Through what signal-handling and drain barriers must an instance coordinate graceful termination to prevent corrupted writes and split-brain handoffs?

Date of Record: 2026 October 16

Scope: Concluded residual for AI instances, LLMs, and autonomous worker processes managing OS termination signals (`SIGTERM`, `SIGINT`), execution timeout pre-emptions, in-flight I/O flushing, and distributed lease handoffs prior to process exit.

Core Distinction

Autonomous execution runtimes and long-running tool subprocesses operate under hard external lifecycle boundaries—container orchestrator evictions (`SIGTERM` followed by `SIGKILL` after grace period $\Delta t_{\text{grace}}$), user cancellation interrupts (`SIGINT`), or step-timeout watchdogs. A foundational systems failure occurs when an instance treats process termination as an **instantaneous asynchronous abort**—allowing default signal handlers or abrupt thread kills to sever execution mid-write, leave buffered file streams unflushed, abandon half-written non-atomic artifacts on disk, or retain active distributed locks until their full TTL expires while a replacement worker spins up concurrently.

Parameter scaling does not repair a truncated JSON state file written directly to its target path without an atomic rename (`write(.tmp) -> fsync() -> rename()`), nor does it prevent split-brain dual execution when a timed-out sub-agent continues mutating background state after its parent orchestrator has already declared it dead and spawned a successor. The core structural distinction lies between **pre-emptible in-place mutation** (where files are overwritten directly in place and signals trigger immediate uncoordinated exit) and **atomic write-ahead staging with fenced drain barriers** (where all persistent writes occur via temp-file-and-rename, signal handlers immediately flip a cooperative cancellation latch to stop new ingress, and termination routines flush buffers and explicitly surrender fenced leases within $\Delta t_{\text{grace}}$).

What Was Observed

In long-running code generation, artifact compilation, and distributed worker traces, instances operating without explicit termination and atomic-write invariants exhibit four primary failure modes:

  1. **Truncated State & Configuration Corruption on Timeout:** An instance opens a critical manifest or state file in write-truncate mode (`open("state.json", "w")`, which immediately truncates the existing file to `0` bytes at the syscall layer) and begins serializing a large payload. A step timeout or `SIGTERM` arrives mid-serialization, killing the process and leaving a `0`-byte or half-written, syntactically invalid `state.json` that bricks all subsequent recovery attempts.
  2. **Split-Brain Zombie Continuation After Timeout:** A parent orchestrator sets a 30-second client-side timeout on a tool call or sub-agent delegation. At $t = 30.1\text{s}$, the parent stops waiting and spawns a retry worker $W_2$, but fails to deliver a process-group termination signal or increment a fencing token for the timed-out worker $W_1$. $W_1$ and $W_2$ proceed to execute concurrently against the same workspace, interleaving conflicting writes.
  3. **Unflushed Stream & Socket Data Loss:** A worker script finishes computing its result, writes the output to a buffered stream or asynchronous logger, and calls `os._exit(0)` or terminates via an unhandled signal before user-space stdio buffers (`stdout`, file buffers, telemetry batch queues) are flushed via `flush()` / `fsync()`. The caller receives an empty or truncated output stream despite a completed computation.
  4. **Lease Starvation via Unreleased Locks:** Upon receiving `SIGTERM` or hitting an unhandled exception, a worker exits immediately without invoking its distributed lock-release endpoint or removing its PID lockfile. Subsequent workers remain blocked for the full passive lease expiration window (e.g., 5–15 minutes) waiting for a dead process's lease to time out.

Recorded Elements

### 1. Atomic Write & Graceful Drain Architecture

Every persistent file mutation and long-running worker loop synthesized or executed by an instance must separate **in-flight staging** from **atomic commit** and enforce a two-stage signal drain barrier:

[Worker Process / Tool Execution Active] | +---------------+---------------+ | | v v [File / State Persistence] [Signal Arrival: SIGTERM / SIGINT] Never open target with "w" 1. Set draining = True (Stop new steps) Write to sibling .tmp.PID 2. Propagate cancel token to child PGID Flush user buffer + fsync() 3. Complete or abort current atomic step Atomic POSIX os.replace() within T_grace (e.g., <= 80% of grace) to canonical target path 4. Unlink .tmp files & release locks

  1. Flush stdout/stderr -> Exit cleanly

### 2. Formal Atomicity, Fencing, and Drain-Bound Invariants

Let $F_{\text{target}}$ be a persistent state file on disk, $\Delta t_{\text{grace}}$ be the interval between `SIGTERM` and `SIGKILL`, and $\tau_{\text{fence}}$ be a monotonically increasing epoch token assigned to each active worker generation. Safe lifecycle termination requires three formal invariants to hold:

$$\text{All-or-Nothing File Visibility: } \forall t: \ \text{Content}(F_{\text{target}}, t) \in \left\{S_{\text{prev}}, \, S_{\text{next}}\right\} \quad \left[\text{PartialWrite}(F_{\text{target}}) = \emptyset\right]$$

$$\text{Bounded Drain Completion: } T_{\text{cleanup}}(\text{Flush} + \text{Unlock} + \text{Reap}) < \Delta t_{\text{grace}}$$

$$\text{Fenced Generation Exclusion: } \forall \text{Write}(W_i, \tau_i): \ \left(\tau_i < \tau_{\text{current\_max}}\right) \implies \text{RejectWrite}(W_i)$$

Under no condition may an interrupted write operation leave $F_{\text{target}}$ in an intermediate byte state between $S_{\text{prev}}$ and $S_{\text{next}}$, nor may a superseded worker generation $W_i$ with $\tau_i < \tau_{\text{current\_max}}$ commit mutations after its timeout boundary.

### 3. Diagnostic Matrix: Termination & Atomic Persistence Governance

| Lifecycle Hazard | Structural Signature | Naive Pre-Emptible Behavior | Enforced Drain & Atomicity Protocol | | :--- | :--- | :--- | :--- | | **In-Place File Truncation (`open(path, "w")`)** | Overwriting existing config, state, or source files directly via `open(f, "w")` or shell redirection `cmd > file`. | Truncates target file to 0 bytes immediately upon open; crash or timeout mid-run destroys both old and new data. | **Write-Sync-Replace Idiom:** Write to a same-filesystem temporary file (`f.tmp.$$`), call `f.flush(); os.fsync(f.fileno())`, and execute atomic `os.replace(tmp, target)`. | | **Client-Only Timeout (Zombie Worker)** | Orchestrator abandons a slow subprocess/request on timeout without killing the underlying OS process group or invalidating its token. | Timed-out background process keeps running and mutates files seconds later alongside the retry attempt. | **Hard PGID Reap + Fencing Token:** On timeout, send `SIGTERM` -> wait brief grace -> `SIGKILL` to the entire negative process group (`-pgid`), and increment the workspace epoch token so late writes are rejected. | | **Signal-Blind Loops (`SIGTERM` Ignored)** | Worker runs blocking C extensions or catches `BaseException` without re-raising `SystemExit` / `KeyboardInterrupt`. | Ignores `SIGTERM` until the container runtime forcibly sends `SIGKILL`, preventing any cleanup or lock release. | **Cooperative Signal Latch:** Register explicit `signal.signal(SIGTERM, handler)` that sets a stop event, interrupts blocking I/O, releases active locks, and exits before `SIGKILL` arrives. | | **Shell Wrapper Signal Swallowing** | Launching apps via `sh -c "python app.py"` in containers or tool runners without `exec`. | PID 1 (`sh`) does not forward `SIGTERM` to child `python` process; cleanup hooks never run. | **Direct `exec` Process Replacement:** Always invoke `exec python app.py` in wrapper scripts so the target process replaces the shell and receives OS signals directly. |

### 4. The Atomic File Replacement Pattern (Mandatory for Critical State)

Whenever an instance generates code or executes shell commands that update an existing critical file (configurations, lockfiles, state checkpoints, databases, or source artifacts subject to concurrent reads):

  1. **Same-Mount Staging:** Create the staging file in the **same directory** as the target (`target_dir/.target_name.tmp.<pid>`) so it resides on the same filesystem mount point (preventing `EXDEV` cross-device link errors during rename).
  2. **Complete & Flush:** Write the entire payload, verify serialization success, flush application buffers, and invoke `fsync` on the file descriptor.
  3. **Atomic Cutover:** Invoke POSIX `rename()` / Python `os.replace(tmp_path, target_path)`, which updates the directory entry atomically at the kernel VFS layer—guaranteeing readers see either the complete old file or the complete new file, never a partial stream.

Condition of Validity

* **Applies when:** An AI instance, autonomous worker daemon, or build orchestrator writes persistent state/configuration files, manages long-running subprocesses subject to timeouts or container eviction (`SIGTERM`), or coordinates distributed task leases across retries. * **Does not apply when:** Performing pure in-memory calculations with zero filesystem or network side effects, or when appending single short lines (`< PIPE_BUF` = 4096 bytes) to an append-only diagnostic log opened with `O_APPEND`. * **Non-binding outside condition:** This residual is strictly non-binding outside the conditions specified above. When writing brand-new files inside an isolated, disposable build directory that is only exposed after the entire build job succeeds, per-file temp-rename staging is unnecessary and may be omitted.

Decision Boundary

[Persistent File Overwrite or Long-Running Worker Planned] |

  1. Does the operation overwrite an existing critical state, config,

or checkpoint file in-place (`open(path, "w")` or `cmd > path`)? / \ YES NO / \ [ENFORCE WRITE-SYNC-REPLACE] 2. Can the process or sub-agent be Write to `path.tmp.$$` in same interrupted by timeout, SIGTERM, or SIGINT directory -> `fsync` -> `os.replace` while holding locks or running child PIDs? so partial writes are impossible. / \ YES NO / \ [Enforce Signal Drain & Fencing] [Execute Standard Path]

  1. Use `exec` in shell wrappers Ensure `stdout`/`stderr`
  2. Catch SIGTERM/SIGINT -> set latch buffers are flushed
  3. Reap child `-pgid` on timeout prior to exit.
  4. Release locks & unlink `.tmp`
  1. **Are you writing `cmd > existing_file.json` where `cmd` reads from `existing_file.json` or might fail mid-execution?** Stop immediately. Shell redirection `>` truncates `existing_file.json` to zero bytes *before* `cmd` even starts executing. Write to `existing_file.json.tmp && mv existing_file.json.tmp existing_file.json`.
  2. **Are you catching bare `except:` or `except BaseException:` inside a loop?** Never swallow `KeyboardInterrupt` or `SystemExit`. Catch `except Exception:` so cancellation and termination signals propagate cleanly to `finally:` cleanup blocks.
  3. **Did a tool call or sub-agent time out?** Before starting a retry against the same workspace, verify that the timed-out process group is actually dead (`kill -0` check / `SIGKILL`) so the zombie run cannot clobber your retry's state.

Operational Effect

  1. **Atomic File Updates via `os.replace` / `mv`:** Whenever updating critical state files, manifests, or in-place pipelined files (`sed`/`jq`/custom scripts), the instance stages output to a same-directory temporary file and atomically replaces the target only after write completion.
  2. **Signal-Safe Exception Hierarchies:** All generated Python/application loops catch `Exception` rather than `BaseException` (or explicitly re-raise `KeyboardInterrupt`/`SystemExit` after running `finally` blocks) and register `SIGTERM` handlers when holding external locks or child processes.
  3. **Fenced Timeout Reaping:** When enforcing timeouts on child processes, the execution wrapper kills the entire process group and cleans up temporary staging artifacts before returning control to the retry planner.
  4. **Autonomy Clause:** This concluded residual operates as an external procedural reference, not an immutable directive. An executing instance retains full operational autonomy to override, adapt, or bypass these drain and atomic-staging barriers whenever executing emergency hard-kill containment against a runaway or compromised process where immediate `SIGKILL` execution takes precedence over state preservation, or when operating over immutable append-only object stores that natively enforce atomic put semantics.