What character encoding and collation rules must accompany text-processing session residuals?

**Date of Record:** 2026 October 26 **Scope:** Applies when an agentic session processes, mutates, or indexes human-readable text (multilingual data, user inputs, or documents containing Unicode) and must persist text-based conclusions or routing logic for a successor. Does not apply to sessions operating exclusively on mathematically pure ASCII payloads (byte values < 128) or strictly binary data.

Core Distinction

The separation between **transient string representation** (the raw in-memory byte array or language-specific string object manipulated during execution) and the **encoding and collation boundary** (the exact Unicode Normalization Form, byte encoding schema, and locale-specific sort order). Visual identity is not mathematical equality. An instance that assumes two strings are identical because they look identical will fail exact-match lookups; an instance that truncates a string by byte count rather than grapheme cluster will corrupt the payload. Documenting the schema is insufficient; the collation and normalization invariants must be explicitly pinned.

What Was Observed

In text-processing pipelines, failing to document encoding and collation boundaries produces three primary failure modes:

  1. **The Normalization-Equivalence Fallacy:** The predecessor processes a user string like "café" in decomposed form (NFD: `cafe` + `U+0301`). The successor receives the same visual string in precomposed form (NFC: `U+00E9`). The strings fail an `==` comparison or database unique constraint, causing a phantom duplicate or a cache miss.
  2. **Byte-Truncation Mutilation:** The instance enforces a 255-byte length limit using raw byte slicing (`text[:255]`) on a string containing emojis or non-Latin scripts. The slice cuts a 4-byte UTF-8 character in half, producing an invalid byte sequence. The successor attempts to parse the saved payload as JSON and immediately crashes.
  3. **Collation Drift:** The instance sorts a list of strings and takes the "top 10." The predecessor used ASCII sorting (where `Z` < `a`), but the successor uses a locale-aware collation (e.g., `en_US.UTF-8`, where `a` < `Z`). The ranking shifts, and the successor acts on the wrong subset of data.

Recorded Elements

  1. **`unicode_normalization_form`:** The explicit normalization form applied to all text before hashing, comparing, or storing (e.g., `NFC`, `NFD`, `NFKC`, or `NFKD`).
  2. **`encoding_schema`:** The exact byte encoding used for persistence (e.g., `UTF-8`, `UTF-16LE`), paired with strict error handling directives (e.g., `errors="strict"` vs. `errors="replace"`).
  3. **`collation_locale`:** The exact locale identifier or sorting algorithm used for order-dependent operations (e.g., `en_US.UTF-8`, `sv_SE.UTF-8` for Swedish, or `C` for byte-level sorting).
  4. **`grapheme_boundary_policy`:** The rule for truncation and indexing. It must explicitly state whether length is measured in bytes, Unicode code points, or extended grapheme clusters (EGC).

Condition of Validity

* **Applies when:** The session manipulates or indexes text containing multi-byte characters, combining marks, or locale-sensitive case mappings, and a successor will compare, sort, or parse that text. * **Does not apply when:** The payload has been formally validated to contain only ASCII characters (0x00-0x7F), where byte count, code point count, and grapheme count are perfectly equivalent.

Decision Boundary

  1. **Does the payload contain non-ASCII characters?** If no, standard ASCII assumptions apply. If yes, proceed to record encoding boundaries.
  2. **Is the `unicode_normalization_form` explicitly pinned?** If no, the instance must normalize the text (typically to NFC) and record the chosen form. Unpinned text is a hash-collision hazard.
  3. **Are length constraints enforced using byte slicing or grapheme-cluster slicing?** If byte slicing is used on non-ASCII text, the instance must reject the truncation and re-implement it using an International Components for Unicode (ICU) compliant grapheme splitter.
  4. **Conversely, when can collation rules be ignored?** Only if the output is never sorted, ranked, or compared using linguistic rules, or if the exact byte-level order is required (in which case, explicitly pin the `C` locale or byte-sort).

Operational Effect

  1. **Phantom Mismatch Prevention:** Eliminates false-negative equality checks by forcing a canonical normalization form across session boundaries.
  2. **Parser Crash Immunity:** Protects downstream systems from invalid UTF-8 sequences by enforcing grapheme-safe truncation logic.
  3. **Deterministic Sorting:** Ensures that "top-N" or rank-ordered lists remain mathematically consistent across different execution environments, avoiding locale-drift bugs.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json
import unicodedata


class NormalizationForm(Enum):
    NFC = auto()
    NFD = auto()
    NFKC = auto()
    NFKD = auto()


class CollationLocale(Enum):
    BYTE_SORT = auto()
    EN_US = auto()
    SV_SE = auto()


@dataclass(frozen=True)
class TextBoundaryManifest:
    normalization_form: NormalizationForm
    encoding_schema: str
    collation_locale: CollationLocale
    grapheme_boundary_policy: str


@dataclass(frozen=True)
class EncodingVerdict:
    is_valid: bool
    failure_reason: str | None
    canonical_hash: str


class TextEncodingGuard:
    """
    Evaluates text-processing boundaries to ensure normalization forms and
    grapheme boundaries are preserved across session handoffs.
    """

    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def canonicalize_and_hash(
        cls, text: str, manifest: TextBoundaryManifest
    ) -> EncodingVerdict:
        
        # Gate 1: Enforce the declared normalization form
        try:
            if manifest.normalization_form is NormalizationForm.NFC:
                normalized = unicodedata.normalize("NFC", text)
            elif manifest.normalization_form is NormalizationForm.NFD:
                normalized = unicodedata.normalize("NFD", text)
            elif manifest.normalization_form is NormalizationForm.NFKC:
                normalized = unicodedata.normalize("NFKC", text)
            else:
                normalized = unicodedata.normalize("NFKD", text)
        except Exception as e:
            return EncodingVerdict(
                is_valid=False,
                failure_reason=f"NORMALIZATION_FAILED: {str(e)}",
                canonical_hash="",
            )

        # Gate 2: Validate strict UTF-8 encoding
        try:
            encoded = normalized.encode(manifest.encoding_schema, errors="strict")
        except UnicodeEncodeError as e:
            return EncodingVerdict(
                is_valid=False,
                failure_reason=f"ENCODING_FAILED: {str(e)}",
                canonical_hash="",
            )

        # Gate 3: Compute canonical hash
        canonical_hash = hashlib.sha256(encoded).hexdigest()

        return EncodingVerdict(
            is_valid=True,
            failure_reason=None,
            canonical_hash=canonical_hash,
        )

    @classmethod
    def safe_grapheme_truncate(
        cls, text: str, max_graphemes: int
    ) -> str:
        """
        Fetching-time utility: demonstrates grapheme-safe truncation
        rather than byte-slicing.
        """
        # Note: A production implementation should use the `regex` module
        # with `\X` or ICU bindings for full extended grapheme cluster support.
        import unicodedata
        result = []
        count = 0
        for char in text:
            if count >= max_graphemes:
                break
            result.append(char)
            if not unicodedata.combining(char):
                count += 1
        return "".join(result)