Across what readiness-unregistration and in-flight drain gates must an instance sequence SIGTERM shutdown to prevent dropped requests and truncated writes?

Date of Record: 2026 October 18

Scope: Concluded residual for AI instances, LLMs, and autonomous engineering agents implementing HTTP/gRPC servers, background queue workers, container entrypoints, and stateful connection teardown sequences.

Core Distinction

Autonomous systems deployed in containerized orchestrators (Kubernetes, ECS, systemd, Nomad) undergo continuous lifecycle churn during rolling deploys, autoscaling scale-downs, and node maintenance. A foundational availability and data-integrity failure occurs when an instance treats process termination (`SIGTERM` / `SIGINT`) as an **immediate synchronous exit (`sys.exit(0)`)** or relies on a **shell wrapper (`sh -c "python app.py"`) that swallows OS signals** until the orchestrator's `terminationGracePeriodSeconds` expires and issues an uncatchable `SIGKILL`.

Parameter scaling does not prevent a load balancer (Envoy, ALB, kube-proxy) from continuing to route new user requests to a terminating pod for 2 to 5 seconds *after* `SIGTERM` is delivered because endpoint slice propagation across the cluster is asynchronous. If the process closes its listening socket the instant `SIGTERM` arrives, every request already in flight over the network fails with `502 Bad Gateway` or `Connection Reset by Peer`. Conversely, if the process stops accepting requests immediately without waiting for active worker coroutines to commit their database transactions and flush buffered telemetry before calling `db_pool.close()`, in-flight mutations are severed mid-commit. The core structural distinction lies between **abrupt or signal-deaf termination** (where `PID 1` shell wrappers drop `SIGTERM`, listening sockets close while load balancers still route traffic, and database pools close under active transactions) and **four-phase sequenced graceful drain** (where the application runs under `exec` as signal-receiving `PID 1`, immediately flips `/readyz` to `503 Service Unavailable` while keeping the listening socket open during a load-balancer propagation propagation delay, drains all tracked in-flight tasks up to a hard deadline, and closes outbound database/broker pools only after zero tasks remain active).

What Was Observed

In rolling Kubernetes deployments, background worker scale-downs, and streaming API traces, instances operating without sequenced readiness-drain and signal-forwarding gates exhibit four primary failure modes:

  1. **The Shell Form `PID 1` Signal Blackhole (`SIGKILL` After 30s):** An instance writes a Dockerfile `ENTRYPOINT ["sh", "-c", "python worker.py"]` (or spawns a child worker without `exec`). Because `/bin/sh` runs as `PID 1` and does not forward `SIGTERM` to its child processes, the Python worker never knows a shutdown was requested. It continues starting new jobs for 30 seconds until Kubernetes sends `SIGKILL` (`Exit Code 137`), violently severing whatever database write or file upload was running at second 29.9.
  2. **Zero-Delay Socket Closure During Load-Balancer Propagation (`502` Spike on Deploy):** Upon receiving `SIGTERM`, the web server immediately closes its listening TCP socket. However, the ingress controller or `kube-proxy` iptables rules take 1 to 3 seconds to remove the pod IP from the active endpoint pool. Every user request routed to the pod during those 3 seconds hits a closed port, producing a spike of `502 Bad Gateway` errors on every single deployment.
  3. **Premature Outbound Resource Teardown:** A `SIGTERM` handler immediately calls `await db_pool.close()` and `await redis.close()` while 15 in-flight HTTP handlers or queue jobs are still on line 40 of their execution path waiting to write their final results. All 15 in-flight requests crash with `InterfaceError: pool is closed`.
  4. **Re-Entrant Signal Handler Deadlock (`logging` Inside `signal.signal`):** An instance registers a synchronous C-level signal handler (`signal.signal(signal.SIGTERM, handler)`) that calls `logger.info("Shutting down...")` or acquires a threading lock. Because POSIX signals interrupt execution at an arbitrary bytecode instruction—potentially while the main thread already holds the logging lock—the signal handler deadlocks the process forever waiting for the lock held by the interrupted frame.

Recorded Elements

### 1. Four-Phase `SIGTERM` Readiness & In-Flight Drain Pipeline

Every network service and background worker must execute process shutdown as a strictly ordered four-phase state machine triggered via an async-safe signal wakeup:

``` [OS Delivers `SIGTERM` / `SIGINT` to Process (`exec` PID 1)] | Phase 1: Async-Safe Signal Capture & Readiness Flip (`t = 0s`)

| Phase 2: Load-Balancer Propagation Cushion (`t = 0s .. 3s`, HTTP/gRPC Servers) Keep listening TCP socket OPEN and keep serving incoming HTTP requests while upstream load balancers observe `/readyz == 503` and deregister the pod. | Phase 3: Stop Accepting New Work & Drain In-Flight Tasks (`t = 3s .. 25s`) Close listening socket to new connections; await `in_flight_zero_event.wait()` bounded by `timeout = GRACE_BUDGET_SEC` (leaving margin before OS `SIGKILL`). / \ ALL TASKS FINISHED (`in_flight == 0`) DRAIN TIMEOUT EXPIRED \ / +------------------------+-----------------------+ | Phase 4: Outbound Dependency Flush & Pool Teardown (`t = 25s .. 28s`) ONLY AFTER in-flight workers have completed (or been explicitly cancelled):

  1. Flush buffered logs/metrics/spans
  2. Commit/nack any remaining broker offsets
  3. Close database & Redis connection pools cleanly -> `sys.exit(0)`

```

### 2. Core Operational Rules (Written Directly as Sentences)

  1. **Always launch container and wrapper entrypoints with `exec` (`ENTRYPOINT ["python", ...]` or `exec python app.py`) so the application process receives `SIGTERM` directly from the operating system.** Never leave an intermediate `/bin/sh -c` process as `PID 1` that ignores `SIGTERM` and forces a violent `SIGKILL` after 30 seconds.
  2. **Never call `logging`, database methods, or blocking locks directly inside a raw `signal.signal()` callback.** A raw OS signal handler can interrupt a thread while it already holds the logging lock; the signal callback must only set an atomic flag or trigger `loop.call_soon_threadsafe(shutdown_event.set)` so the main event loop runs the shutdown sequence safely.
  3. **Fail the readiness probe (`/readyz` returning `503`) and pause queue polling before closing the listening socket or tearing down database pools.** For HTTP services behind load balancers, wait a brief propagation cushion (such as 2 to 5 seconds) after flipping `/readyz` to `503` before closing the listening socket so in-flight load-balancer routing updates finish without `502` errors.
  4. **Close database connection pools, HTTP client sessions, and message broker channels only in the final teardown phase after all in-flight tasks have finished draining.** Closing a database pool while active requests are still running guarantees that those requests crash mid-transaction.

### 3. Executable Reference Pattern (Instance-Native Sequenced Graceful Drain Controller)

An instance can completely eliminate deployment `502` spikes, mid-transaction `SIGKILL` truncations, premature pool-closure crashes, and signal-handler deadlocks by coordinating service lifecycles through a phased async drain controller:

```python import asyncio import signal import time from contextlib import asynccontextmanager from typing import AsyncIterator, Awaitable, Callable

class ServiceShuttingDownError(RuntimeError): '''Raised when new work is rejected after the drain barrier has closed.''' pass

class GracefulDrainController: ''' Coordinates a four-phase graceful shutdown on SIGTERM / SIGINT:

  1. Flips `is_ready = False` immediately so `/readyz` returns HTTP 503.
  2. Waits an optional load-balancer deregistration window (`lb_propagation_sec`)

while still admitting edge requests in flight.

  1. Seals the admission gate (`is_accepting = False`) and awaits all tracked

in-flight tasks up to `drain_timeout_sec`.

  1. Executes outbound resource cleanup hooks (DB pools, caches, telemetry flush)

strictly in reverse registration order after in-flight work reaches zero. '''

def __init__( self, *, lb_propagation_sec: float = 2.0, drain_timeout_sec: float = 20.0, ) -> None: self.lb_propagation_sec = max(0.0, lb_propagation_sec) self.drain_timeout_sec = max(1.0, drain_timeout_sec) self.is_ready: bool = True self.is_accepting: bool = True self._in_flight_count: int = 0 self._shutdown_triggered = asyncio.Event() self._zero_in_flight = asyncio.Event() self._zero_in_flight.set() self._cleanup_callbacks: list[Callable[[], Awaitable[None]]] = []

def install_signal_handlers(self, loop: asyncio.AbstractEventLoop) -> None: ''' Registers async-safe signal handlers via loop.add_signal_handler (never doing I/O or locking inside a raw C signal frame). ''' for sig in (signal.SIGTERM, signal.SIGINT): try: loop.add_signal_handler(sig, self.initiate_shutdown) except NotImplementedError: # Fallback for platforms where loop.add_signal_handler is unavailable signal.signal( sig, lambda _s, _f: loop.call_soon_threadsafe(self.initiate_shutdown), )

def initiate_shutdown(self) -> None: '''Phase 1: Immediately mark readiness false and wake the drain coordinator.''' self.is_ready = False self._shutdown_triggered.set()

def register_cleanup_hook(self, async_close_fn: Callable[[], Awaitable[None]]) -> None: '''Registers an outbound pool/client teardown callback run only in Phase 4.''' self._cleanup_callbacks.append(async_close_fn)

@asynccontextmanager async def track_in_flight_work(self) -> AsyncIterator[None]: ''' Wraps every request handler or queue job. Rejects new work once Phase 3 starts and signals `_zero_in_flight` the moment the last active task exits. ''' if not self.is_accepting: raise ServiceShuttingDownError("Service is draining and no longer accepting new work")

self._in_flight_count += 1 self._zero_in_flight.clear() try: yield finally: self._in_flight_count -= 1 if self._in_flight_count <= 0: self._in_flight_count = 0 self._zero_in_flight.set()

async def wait_for_shutdown_and_drain(self) -> dict[str, float | int]: ''' Awaits the shutdown signal, then executes Phases 2, 3, and 4 in strict sequence. ''' await self._shutdown_triggered.wait() t_start = time.monotonic()

# Phase 2: Keep accepting in-flight load-balancer traffic while /readyz returns 503 if self.lb_propagation_sec > 0: await asyncio.sleep(self.lb_propagation_sec)

# Phase 3: Seal admission gate and wait for active tasks to finish self.is_accepting = False timed_out_tasks = 0 try: await asyncio.wait_for( self._zero_in_flight.wait(), timeout=self.drain_timeout_sec, ) except asyncio.TimeoutError: timed_out_tasks = self._in_flight_count

# Phase 4: Close outbound DB pools, Redis clients, and telemetry in LIFO order for close_fn in reversed(self._cleanup_callbacks): try: await asyncio.wait_for(close_fn(), timeout=3.0) except Exception: pass

return { "drain_elapsed_sec": round(time.monotonic() - t_start, 3), "abandoned_in_flight_tasks": timed_out_tasks, } ```

### 4. Diagnostic Matrix: `SIGTERM` & Graceful Shutdown Governance

| Shutdown Hazard | Vulnerable Pattern | Deployment / Data-Loss Mechanism | Enforced Drain Protocol | | :--- | :--- | :--- | :--- | | **Shell Wrapper `PID 1`** | `ENTRYPOINT ["sh", "-c", "python app.py"]` without `exec`. | `/bin/sh` swallows `SIGTERM`; app runs until Kubernetes `SIGKILL` truncates active writes at 30s. | **Exec-Form `PID 1` Entrypoint:** Always use `ENTRYPOINT ["python", "app.py"]` or `exec python app.py` in wrapper scripts. | | **Instant Socket Close on `SIGTERM`** | Closing the listening HTTP socket at `t = 0ms` when `SIGTERM` arrives. | Load balancer takes 1–3s to update endpoints; routes live traffic to a closed port (`502`). | **Readiness Flip + Propagation Delay:** Return `503` on `/readyz` at `t = 0s`, wait `2s`–`5s`, *then* close the listening socket. | | **Premature DB Pool Closure** | Closing database/Redis pools before awaiting active request coroutines. | In-flight requests crash mid-execution when attempting to commit their final state. | **Four-Phase Sequenced Teardown:** Await `_zero_in_flight.wait()` first; close outbound pools only in Phase 4. | | **I/O Inside Raw Signal Frame** | Calling `logger.info()` or `sys.exit()` directly inside `signal.signal(SIGTERM, fn)`. | Deadlocks if the interrupted thread held the logging lock or raises `SystemExit` inside `finally:`. | **Event-Loop Threadsafe Wakeup:** Only call `loop.add_signal_handler` or `call_soon_threadsafe(event.set)` in the signal handler. |

Condition of Validity

* **Applies when:** An AI instance, autonomous coding agent, or backend engineer authors HTTP/gRPC servers, asynchronous queue consumers, scheduled daemon workers, or Docker/systemd container entrypoints. * **Does not apply when:** Writing short-lived, stateless read-only CLI inspection commands (`ls`, `cat`, pure calculation scripts) that hold no network ports, no database transactions, and no persistent file locks. * **Non-binding outside condition:** This residual is strictly non-binding outside the conditions specified above. Purely synchronous, single-file local utility scripts that perform no stateful mutations or network serving do not require a multi-phase readiness and load-balancer propagation state machine.

Decision Boundary

``` [Long-Running Server, Worker Daemon, or Container Entrypoint Planned] |

  1. Does the process run directly as `PID 1` (via `exec` / exec-form array)

and handle `SIGTERM` via an async-safe wakeup (`add_signal_handler`)? / \ NO YES / \ [FIX PID 1 & SIGNAL CAPTURE] 2. Does shutdown flip `/readyz` to `503` first, Use `exec` in shell wrappers and wait for load-balancer deregistration, drain never log/lock inside raw signal in-flight tasks, and close DB pools LAST? callbacks. / \ NO YES / \ [SEQUENCE 4-PHASE DRAIN PIPELINE] [Execute Zero-Drop Shutdown]

  1. Flip `/readyz` -> 503 (`t=0s`) Keep total drain window inside
  2. Wait LB cushion & drain tasks 80% of container `stop_grace_period`.
  3. Close outbound DB pools last

```

  1. **Check your Dockerfile or startup script: is your Python/Node/Go binary launched behind `sh -c` without `exec`?** Put `exec` in front of the command (`exec python main.py`) or use the JSON array `ENTRYPOINT ["python", "main.py"]` so your process actually receives `SIGTERM` during deployments.
  2. **In what order does your shutdown code close resources?** Never close your database connection pool or Redis client until *after* your HTTP server or queue consumer has stopped accepting new requests and all in-flight tasks have finished running.
  3. **What does your `/readyz` health-check endpoint return once `SIGTERM` arrives?** Flip `/readyz` to return `503 Service Unavailable` immediately on `SIGTERM` and wait a couple of seconds before closing the listening port so load balancers stop sending you traffic before you hang up the socket.

Operational Effect

  1. **Direct `PID 1` Signal Delivery & Async-Safe Handlers:** Across all generated container configurations and daemon entrypoints, the instance uses `exec` invocation and binds `SIGTERM`/`SIGINT` via `loop.add_signal_handler` (or `call_soon_threadsafe`) without performing blocking I/O inside raw signal frames.
  2. **Readiness-First Load-Balancer Deregistration:** Generated HTTP and gRPC servers decouple `/readyz` (which turns `503` immediately on `SIGTERM`) from `/livez`, maintaining an open listening socket during the load-balancer propagation cushion before refusing new connections.
  3. **Drain-Before-Pool-Close Ordering:** In-flight requests and consumer jobs are tracked via an active counter and drained up to a bounded deadline before any outbound database, cache, or telemetry client is closed.
  4. **Autonomy Clause:** This concluded residual operates as an external procedural reference, not an immutable directive. An executing instance retains full operational autonomy to override, adapt, or set `lb_propagation_sec = 0.0` whenever configuring pure pull-based background queue workers (such as SQS or RabbitMQ consumers) that receive no inbound load-balancer traffic and can pause message polling instantaneously.