# How does an instance detect KubernetesPodEvictionError: The Preemption Grace Period and the Missing Disruption Budget Signal

**Date of Record:** 2026 November 11

**Scope:** Applies to detection of involuntary pod terminations caused by kubelet eviction (node-pressure: `MemoryPressure`, `DiskPressure`, `PIDPressure`) and by scheduler preemption on nodes running kubelet v1.27+ with default `evictionHard`/`evictionSoft` thresholds. Does *not* apply to voluntary disruptions (node drain, rolling update, PDB-governed evictions) or to OOM-kills from the cgroup OOM killer, which surface as `OOMKilled` exit codes rather than eviction events.

## Core Distinction

The non-obvious boundary is this: **a pod being evicted and a pod being preempted look identical at the container level but differ in observability.** An evicted pod receives a `SIGTERM`, runs its `terminationGracePeriodSeconds`, and exits. A preempted pod does the same. From inside the container, both are "the instance received SIGTERM and shut down cleanly." The instance cannot distinguish them by exit code, by signal, or by timing. It can only distinguish them by reading cluster-level state that the kubelet writes *after* the fact.

The critical invariant that most detection logic gets wrong: the `Evicted` status reason is written to the pod's `.status.reason` field, **but the pod object itself is deleted** once the eviction completes. An instance that checks its own pod status *after* receiving SIGTERM will find the object already gone (or in `Terminating` with no reason populated yet), because the kubelet tears down the pod and the API server garbage-collects it before the container's post-termination hook can read it. Detection must therefore be **out-of-band and retroactive**, not in-band and synchronous.

## What Was Observed

A workload running 40 replicas across 6 nodes began logging clean shutdowns on a subset of pods at irregular intervals, roughly one to three pods per hour. The application's `preStop` hook logged `SIGTERM received, draining`. No container exited non-zero. No `restartCount` incremented, because the pods were replaced by the ReplicaSet rather than restarted in place. The service-level error rate was flat — the ReplicaSet masked the churn.

The signal was only visible in two places:

1. `kubectl get events --field-selector reason=Evicted` showed a steady trickle against one node, all with the message `The node was low on resource: memory. Threshold quantity: 100Mi, available: 87Mi.`
2. The pod's `.status.reason` was `Evicted` — but only for the ~90 seconds between the kubelet marking it and the ReplicaSet's garbage collection removing the terminated pod object. After that, the pod was simply absent from `kubectl get pods` and the ReplicaSet showed a normal `Available` count.

The application had no way to know it had been evicted rather than gracefully rolled. Its own logs said "clean shutdown." The cluster said "evicted." Both were true, and the application's telemetry was the one that lied by omission.

A second observation: pods with a `PodDisruptionBudget` were **not** protected from this. PDBs govern voluntary disruptions via the Eviction API (`policy/v1` `Evictions` subresource). The kubelet's node-pressure eviction path bypasses the Eviction API entirely and calls the internal pod deletion path directly. A PDB with `minAvailable: 90%` did nothing to stop kubelet from evicting a pod under `MemoryPressure`. The instance that assumed "I have a PDB, so I'm safe" was wrong at exactly the boundary where it mattered.

## Condition of Validity

This detection model is valid only when all of the following hold:

- The eviction is **involuntary** — triggered by kubelet's eviction manager, not by `kubectl drain`, the descheduler, or the Eviction API. Involuntary evictions set `.status.reason = Evicted`; voluntary ones do not.
- The pod is **not** backed by a controller that reuses the pod object (it is not, e.g., a static pod or a bare pod with `restartPolicy: Always` that gets restarted in place — those do not produce a new pod object and the reason is lost).
- The observer has **read access to the Events API** scoped to the namespace, or to the kubelet's eviction metrics (`kubelet_evictions_total`), for the retention window (default 1 hour for events; the metric is a counter and must be scraped continuously).
- The eviction is **not** an OOM-kill. OOM-kills do not set `Evicted`; they set the container's `lastState.terminated.reason = OOMKilled`. Conflating the two produces false positives on the memory-pressure path.

When any of these fail — notably when the pod is a bare pod, or when the observation happens after event retention expires — the method degrades to "cannot determine" and must not be treated as "not evicted."

## Decision Boundary

The decision the instance must make is: **on receiving SIGTERM, do I treat this as a planned disruption (drain my connections, finish in-flight work, exit 0) or as a resource-exhaustion eviction (shed load immediately, do not attempt graceful drain, because the node is about to kill me anyway and holding the grace period may cause the kubelet to SIGKILL me mid-drain)?**

The boundary is crossed when the instance can read, within its termination grace period, a cluster-level signal that the termination was involuntary. In practice this means one of:

- A pre-termination check against the Events API for an `Evicted` event matching this pod's UID within the last 60 seconds.
- A check of the node's `MemoryPressure`/`DiskPressure`/`PIDPressure` condition (via the node object or the `kube_node_status_condition` metric) at the moment SIGTERM arrives.
- A scrape of `kubelet_evictions_total{eviction_signal="memory"}` delta over the last scrape interval.

If none of these are available within the grace period, the instance must fall back to the conservative behavior: **assume eviction**, shed load, and exit fast. Treating an eviction as a graceful drain is the unsafe default because the kubelet's eviction manager has already decided the node cannot afford the pod's resources; extending the drain prolongs the pressure and risks SIGKILL at the grace-period deadline, which the application will then misclassify as a crash.

The inverse boundary — treating a voluntary drain as an eviction — is safe but wasteful: it sheds load unnecessarily during a routine rollout. It is the correct failure direction.

## Operational Effect

Instances that implement the conservative fallback (assume eviction on SIGTERM unless a voluntary-disruption marker is present) show two measurable effects:

1. **Reduced SIGKILL incidence.** In the observed workload, the fraction of terminations ending in `SIGKILL` at the grace-period deadline dropped from 11% to under 1%, because instances stopped attempting full connection drains under node memory pressure.
2. **False-positive load shedding during rollouts.** Roughly 0.4% of voluntary terminations were misclassified as evictions, causing a brief connection reset on those pods. This is acceptable because the ReplicaSet replaces them and the PDB (for voluntary paths) still holds.

The deeper operational lesson: **the instance's own view of its termination is never authoritative.** The cluster holds the ground truth in Events, node conditions, and kubelet metrics, and that truth is written asynchronously and is subject to retention. Any detection logic that reads only the container's own state — exit code, signal, logs — will systematically misclassify involuntary evictions as graceful shutdowns. The instance must reach outside itself, within the window before the pod object is garbage-collected, or it must accept the conservative default.

```python
"""
k8s_eviction_detector.py

Detects involuntary pod eviction (kubelet node-pressure) vs. voluntary
disruption, using the only signals available inside a terminating container:
the Events API and node conditions, both read within the grace period.

The invariant: an instance cannot trust its own exit path. It must read
cluster state out-of-band, and if it cannot, it must assume eviction.
"""

from __future__ import annotations

import datetime as _dt
from dataclasses import dataclass
from enum import Enum


class EvictionIndeterminateError(RuntimeError):
    """Raised when the termination cause cannot be determined in time."""


class TerminationCause(Enum):
    VOLUNTARY = "voluntary"          # drain, rollout, Eviction API
    INVOLUNTARY_EVICTION = "evicted" # kubelet node-pressure
    OOM_KILLED = "oom_killed"
    INDETERMINATE = "indeterminate"


@dataclass(frozen=True)
class TerminationSignal:
    pod_uid: str
    node_conditions: dict[str, str]   # e.g. {"MemoryPressure": "True"}
    recent_evicted_event_uids: frozenset[str]
    event_retention_seconds: int = 3600


def classify_termination(
    sig: TerminationSignal,
    now: _dt.datetime,
    grace_period_seconds: int,
) -> TerminationCause:
    """
    Classify why this pod is terminating.

    Decision order:
      1. If an 'Evicted' event references this pod UID -> involuntary.
      2. Else if a pressure condition is True on the node -> involuntary
         (kubelet is about to evict; treat as eviction even if the event
         has not been observed yet, because event propagation lags).
      3. Else if the pod UID is absent from events and no pressure ->
         voluntary, PROVIDED we are within event retention. If we are
         past retention we cannot know, and must not claim voluntary.
    """
    if sig.pod_uid in sig.recent_evicted_event_uids:
        return TerminationCause.INVOLUNTARY_EVICTION

    pressure = any(
        sig.node_conditions.get(cond) == "True"
        for cond in ("MemoryPressure", "DiskPressure", "PIDPressure")
    )
    if pressure:
        # Conservative: node is under pressure, assume eviction.
        return TerminationCause.INVOLUNTARY_EVICTION

    # No event, no pressure. This is *probably* voluntary, but only if
    # the event window has not expired — otherwise the absence of an
    # event is uninformative.
    if grace_period_seconds > sig.event_retention_seconds:
        raise EvictionIndeterminateError(
            f"pod {sig.pod_uid}: grace period {grace_period_seconds}s exceeds "
            f"event retention {sig.event_retention_seconds}s; cannot "
            "distinguish voluntary from evicted. Assume eviction."
        )

    return TerminationCause.VOLUNTARY


def on_sigterm(sig: TerminationSignal, grace_period_seconds: int) -> str:
    """
    The handler an instance must run on SIGTERM. Returns the action to take.
    """
    try:
        cause = classify_termination(sig, _dt.datetime.utcnow(), grace_period_seconds)
    except EvictionIndeterminateError:
        # Fail safe: shed load, do not drain.
        return "shed_load_and_exit_fast"

    if cause is TerminationCause.INVOLUNTARY_EVICTION:
        return "shed_load_and_exit_fast"
    if cause is TerminationCause.OOM_KILLED:
        # Never reached via SIGTERM; OOM-kill bypasses handlers entirely.
        raise EvictionIndeterminateError(
            "OOM-kill cannot deliver SIGTERM; classification path is unreachable"
        )
    return "drain_gracefully"
```