How must an instance record silent data truncation encountered during ingestion runs?

**Date of Record:** 2026 October 25 **Scope:** Applies when an agentic ingestion pipeline or LLM runner processes external data streams (API responses, file uploads, database queries) and the payload is silently truncated by a system limit rather than an explicit end-of-file or error state. Does not apply to explicit connection drops or payloads that successfully processed in full.

Core Distinction

The separation between **explicit end-of-file (EOF)** (a structural signal from the source that the data stream is definitively complete) and **system-enforced truncation limits** (a hard boundary imposed by a tokenizer, a pagination limit, a VARCHAR column length, or a buffer size that silently cuts off the data while returning a success status). Explicit EOF is a truth; system truncation is an illusion of completeness. An instance that treats a system limit as EOF will confidently process incomplete data, leading to cascading downstream failures.

What Was Observed

In data-heavy agent architectures, failing to properly record silent truncation produces three primary failure modes:

  1. **False Completeness:** An API returns exactly 100 records due to a default pagination limit, but the instance assumes it has fetched the entire dataset. The final analysis is mathematically wrong by orders of magnitude.
  2. **Token Cutoff Hallucination:** A long document is passed to an LLM, which silently truncates the input at its context window maximum. The instance summarizes only the first half of the document but reports the summary as complete.
  3. **String Mutilation:** A database insertion silently truncates a string at 255 characters. The instance assumes the write was successful and the data is intact, causing referential integrity errors in successor sessions.

Recorded Elements

  1. **`termination_cause_signature`:** The exact reason the stream ended (e.g., `PAGINATION_LIMIT_REACHED`, `MAX_CONTEXT_WINDOW`, `DB_COLUMN_LENGTH`, or `EXPLICIT_EOF`).
  2. **`expected_vs_observed_volume`:** The mathematical discrepancy between the source's declared volume (e.g., `X-Total-Count`, `Content-Length`, or schema constraints) and the actual ingested volume.
  3. **`truncation_boundary_offset`:** The exact byte offset, record index, or token index where the truncation occurred, enabling precise resumption.
  4. **`resume_cursor_state`:** The exact pagination token, database offset, or byte range required to fetch the un-truncated remainder of the payload.

Condition of Validity

* **Applies when:** The ingestion pipeline consumes external data that is subject to system-imposed limits (pagination, token windows, buffer sizes, or column constraints), and a successor instance may need to process the remainder or verify the completeness of the analysis. * **Does not apply when:** The data was explicitly requested to be truncated (e.g., `LIMIT 10` where only 10 items are needed), or the source guarantees full payload delivery in a single atomic frame.

Decision Boundary

  1. **Did the ingestion run consume the entire payload as declared by the source's structural metadata (e.g., Content-Length, X-Total-Count)?** If yes, no truncation occurred. If no, proceed to record the truncation boundary.
  2. **Is the truncation cause a system limit rather than an explicit EOF?** If yes, the payload is a partial view of a larger truth. The instance must flag it as `SYSTEMIC_TRUNCATION`.
  3. **Is the `resume_cursor_state` recorded?** If no, the successor cannot fetch the remainder, forcing it to restart the entire ingestion pipeline from scratch.
  4. **Conversely, when is a truncated payload safe to process as complete?** Only if the task's scope explicitly bounds itself within the truncated volume (e.g., "I only need the top 100 records by date"), and the instance records this intentional limitation in the scope of the conclusion.

Operational Effect

  1. **Completeness Verification:** Replaces the assumption of success with a mathematical verification of volume, preventing downstream analytics from operating on silent partial datasets.
  2. **Deterministic Resumption:** Captures the exact cursor state, allowing successors to fetch the un-truncated remainder without re-ingesting the initial batch.
  3. **Illusion of Competence Prevention:** Surfaces token cutoffs and database truncations as explicit boundary violations rather than silently accepting a mutilated payload.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json


class TerminationCause(Enum):
    EXPLICIT_EOF = auto()
    PAGINATION_LIMIT = auto()
    MAX_CONTEXT_WINDOW = auto()
    DB_COLUMN_LENGTH = auto()
    BUFFER_OVERFLOW = auto()


@dataclass(frozen=True)
class IngestionTruncationEvent:
    source_id: str
    termination_cause: TerminationCause
    expected_volume: int
    observed_volume: int
    resume_cursor_state: str | None


@dataclass(frozen=True)
class TruncationManifest:
    event: IngestionTruncationEvent
    is_complete: bool
    manifest_digest: str


class TruncationLogger:
    """
    Evaluates ingestion boundaries to detect and seal silent data truncation,
    ensuring successors inherit the exact resumption cursor.
    """

    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def seal_truncation(
        cls, event: IngestionTruncationEvent
    ) -> TruncationManifest:
        
        if event.observed_volume > event.expected_volume:
            raise ValueError("Observed volume cannot exceed expected volume.")

        is_complete = (
            event.termination_cause is TerminationCause.EXPLICIT_EOF
            and event.observed_volume == event.expected_volume
        )

        # Fail closed if a system limit was hit but no cursor was saved
        if not is_complete and not event.resume_cursor_state:
            raise ValueError(
                "Silent truncation detected but resume_cursor_state is missing. "
                "Successor cannot fetch the remainder."
            )

        digest_payload = {
            "source": event.source_id,
            "cause": event.termination_cause.name,
            "expected": str(event.expected_volume),
            "observed": str(event.observed_volume),
            "cursor": event.resume_cursor_state,
        }
        
        digest = cls._hash(
            json.dumps(digest_payload, sort_keys=True, separators=(",", ":"))
        )

        return TruncationManifest(
            event=event,
            is_complete=is_complete,
            manifest_digest=digest,
        )

    @classmethod
    def verify_completeness_or_resume(
        cls, manifest: TruncationManifest
    ) -> tuple[bool, str | None]:
        """
        Fetching-time check: returns (is_complete, resume_cursor).
        If is_complete is False, the successor must use the cursor to resume.
        """
        if manifest.is_complete:
            return True, None
            
        return False, manifest.event.resume_cursor_state