Across what unicode normalization and byte-encoding boundaries must an instance canonicalize text identifiers to prevent homoglyph collision and mojibake corruption?

Date of Record: 2026 October 16

Scope: Concluded residual for AI instances, LLMs, and autonomous engineering agents handling user usernames, file paths, cryptographic hash inputs, database lookup keys, and cross-platform text serialization.

Core Distinction

Autonomous systems constantly compare, hash, index, and persist text strings across heterogeneous operating systems, filesystems, and database engines. A foundational data-integrity and security failure occurs when an instance treats **visual glyph identity or raw Python/JavaScript string equality (`a == b`)** as canonical equivalence, ignoring the underlying Unicode normalization form (`NFC` vs. `NFD` vs. `NFKC`), invisible zero-width code points, mixed-script confusable homoglyphs, and implicit OS locale encodings (`cp1252` vs. `utf-8`).

Parameter scaling does not make `"caf\u00e9"` (4 code points, precomposed `NFC` standard on Linux/Windows) compare equal to `"cafe\u0301"` (5 code points, decomposed `NFD` standard on macOS HFS+/APFS) in a byte-wise hash or dictionary lookup, nor does it stop `open("data.json", "r")` without an explicit `encoding="utf-8"` argument from crashing with `UnicodeDecodeError` when run on a Windows worker defaulting to `cp1252`. The core structural distinction lies between **unnormalized locale-dependent string handling** (where visually identical strings produce distinct SHA-256 hashes, bypass uniqueness constraints, or corrupt across OS boundaries) and **explicit UTF-8 and Unicode-normalized canonicalization** (where every text boundary enforces `encoding="utf-8"`, normalizes human text to `NFC` and security/identifier tokens to `NFKC`, strips invisible format characters, and rejects mixed-script homoglyph spoofing before indexing or hashing).

What Was Observed

In cross-platform file synchronization, authentication/account creation, and cryptographic hashing traces, instances operating without explicit encoding and Unicode normalization gates exhibit four primary failure modes:

  1. **`NFC` vs. `NFD` Cross-Platform Filename & Lookup Split:** An instance writes a file synchronization or lookup utility that indexes filenames in a Python dictionary or SQL table. A file named `r\u00e9sum\u00e9.pdf` created on macOS is stored by the filesystem in decomposed `NFD` form (`e` + combining acute accent `\u0301`), while a query from a Linux client or web form sends precomposed `NFC` (`\u00e9`). Although both strings render identically on screen as `résumé.pdf`, `filename_in_db == query_filename` evaluates to `False`, causing duplicate records, missed cache hits, and broken file lookups.
  2. **Account & Identifier Homoglyph Spoofing (`NFKC` / Mixed-Script Bypass):** An instance validates that a newly registered username, domain slug, or package name `"admin"` does not already exist in the database. An attacker registers `"\u0430dmin"` (using Cyrillic small letter `а` `\u0430` instead of Latin `a` `\u0061`) or `"finance"` (using the compatibility ligature `\ufb01`). Because a raw string check sees distinct code points, the system allows a visually indistinguishable impersonator identifier.
  3. **Host-Locale Default Encoding Crash (`cp1252` / `ASCII` Mojibake):** An instance writes `Path("output.json").write_text(data)` or `open("README.md").read()` without specifying `encoding="utf-8"`. The code passes in a Linux container (`LANG=C.UTF-8`), but immediately throws `UnicodeEncodeError: 'charmap' codec can't encode character` or silently corrupts UTF-8 multi-byte sequences into mojibake (`é`) when executed on Windows or inside a minimal cron environment (`LC_ALL=C`).
  4. **BOM (`\ufeff`) & Invisible Zero-Width Character Poisoning:** An instance parses a CSV header or configuration key exported from an external tool that begins with a UTF-8 Byte Order Mark (`\ufeffid`) or contains a trailing zero-width space (`\u200b`). Lookups for `row["id"]` fail with `KeyError: 'id'` even though printing the dictionary keys appears to show `'id'` visually in the terminal.

Recorded Elements

### 1. Decode-Normalize-Scrub-Verify Text Pipeline

Every external string or identifier crossing an I/O boundary must pass through an explicit encoding and Unicode canonicalization pipeline before participating in equality checks, database queries, or cryptographic hashes:

``` [Raw Byte Stream or External String Arrives at Boundary] | Gate 1: Explicit UTF-8 Decoding & BOM Stripping Always specify `encoding="utf-8-sig"` (for files that may have BOM) or `encoding="utf-8"`, never relying on OS `locale.getpreferredencoding()`. | Gate 2: Classify String Role (General Prose vs. Canonical Identifier) / \ [General Text / Filenames] [Identifiers / Usernames / Keys] Apply `unicodedata.normalize("NFC", s)` Apply `unicodedata.normalize("NFKC", s)` Unifies `e + \u0301` -> `\u00e9` across Collapses compatibility ligatures (`fi`->`fi`), macOS, Linux, and Windows. full-width chars, and superscripts. \ / +--------------------+---------------------+ | Gate 3: Invisible & Control Character Scrubbing Strip or reject Unicode category `Cf` (format chars like `\u200b`, `\ufeff`, bidi overrides) and `Cc` control characters. | Gate 4: Single-Script / ASCII-Identifier Enforcement (For Security Keys) For usernames, slugs, and package names: enforce pure ASCII allowlist or reject mixed-script tokens (e.g., Latin + Cyrillic in the same word). ```

### 2. Core Operational Rules (Written Directly as Sentences)

  1. **Always pass `encoding="utf-8"` explicitly to every `open()`, `Path.read_text()`, `Path.write_text()`, and `subprocess.run(..., text=True)` call.** Never rely on the operating system's default locale encoding, which varies across Linux, macOS, Windows, and container minimal environments.
  2. **Normalize general text and filenames to `NFC` (`unicodedata.normalize("NFC", text)`) at ingress before comparing, indexing, or hashing them.** Normalizing to `NFC` ensures that precomposed and decomposed representations of the exact same accented characters produce identical byte sequences and identical SHA-256 digests across all operating systems.
  3. **Normalize identifiers, usernames, configuration keys, and search tokens using `NFKC` (`unicodedata.normalize("NFKC", token).casefold()`), and strip invisible format characters (`Cf`).** Using `NFKC` collapses full-width characters and typographic ligatures into their standard canonical equivalents before uniqueness checks run.
  4. **Forbid mixed-script words (such as mixing Latin and Cyrillic or Greek letters in the same identifier) on security-sensitive names.** If an identifier is meant to be an internal slug, key, or username, restrict it to a strict ASCII character allowlist or enforce single-script Unicode validation so visual homoglyph attacks cannot impersonate existing entities.

### 3. Executable Reference Pattern (Instance-Native Unicode & Identifier Canonicalizer)

An instance can completely eliminate `NFC`/`NFD` lookup splits, BOM `KeyError` mysteries, locale `UnicodeDecodeError` crashes, and homoglyph spoofing by routing strings through deterministic canonicalization helpers:

```python import hashlib import re import unicodedata from pathlib import Path

_ASCII_SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9._-]{0,62}[a-z0-9]$")

def canonicalize_text_nfc(raw_text: str) -> str: ''' Normalizes arbitrary UTF-8 prose or filenames to canonical NFC form, strips leading/trailing UTF-8 BOMs and zero-width format characters, and normalizes CRLF/CR line endings to LF. ''' if not isinstance(raw_text, str): raise TypeError(f"Expected str, got {type(raw_text).__name__}")

# Step 1: Strip UTF-8 BOM if present at start of string cleaned = raw_text.lstrip("\ufeff")

# Step 2: Remove invisible format characters (category 'Cf', e.g., \u200b, \u200e) cleaned = "".join( ch for ch in cleaned if unicodedata.category(ch) != "Cf" )

# Step 3: Normalize line endings and apply canonical NFC composition cleaned = cleaned.replace("\r\n", "\n").replace("\r", "\n") return unicodedata.normalize("NFC", cleaned)

def canonicalize_strict_identifier(raw_id: str) -> str: ''' Canonicalizes a security-sensitive identifier (username, slug, config key) using NFKC compatibility normalization + casefolding, and rejects any invisible, control, or non-ASCII homoglyph characters. ''' if not isinstance(raw_id, str): raise TypeError(f"Expected str identifier, got {type(raw_id).__name__}")

# Gate 1: Reject invisible format or control characters outright (do not silently strip) for ch in raw_id: if unicodedata.category(ch) in ("Cf", "Cc", "Cs"): raise ValueError( f"Rejected identifier containing control or invisible format char U+{ord(ch):04X}" )

# Gate 2: Apply NFKC compatibility decomposition/composition + Unicode casefold normalized = unicodedata.normalize("NFKC", raw_id.strip()).casefold()

# Gate 3: Enforce strict ASCII identifier grammar (blocks Cyrillic/Greek homoglyphs) if not _ASCII_SLUG_RE.match(normalized): raise ValueError( f"Identifier {raw_id!r} (normalized: {normalized!r}) violates strict ASCII slug rules " "or contains non-Latin confusable characters." )

return normalized

def hash_canonical_string(text: str) -> str: ''' Computes a platform-independent SHA-256 digest over NFC-normalized UTF-8 bytes so macOS (NFD) and Linux/Windows (NFC) strings yield the exact same hash. ''' canonical = canonicalize_text_nfc(text) return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

def read_utf8_clean(path: str | Path) -> str: '''Reads a text file safely using utf-8-sig (stripping any BOM) and normalizes to NFC.''' raw = Path(path).read_text(encoding="utf-8-sig") return canonicalize_text_nfc(raw) ```

### 4. Diagnostic Matrix: Encoding & Unicode Canonicalization Governance

| Text / Encoding Hazard | Vulnerable Code Signature | Corruption / Exploit Mechanism | Enforced Canonicalization Protocol | | :--- | :--- | :--- | :--- | | **Implicit Locale File I/O** | `open("file.txt").read()` or `Path(p).write_text(s)` without `encoding=`. | Uses OS locale (`cp1252` on Windows, `ASCII` in minimal containers), crashing or corrupting UTF-8 bytes. | **Mandatory Explicit `encoding="utf-8"`:** Specify `encoding="utf-8"` on all writes and `encoding="utf-8-sig"` when reading external CSV/JSON/text files. | | **Cross-OS `NFC` vs. `NFD` Mismatch** | Comparing or hashing filenames/strings directly (`s1 == s2`, `sha256(s.encode())`). | macOS `NFD` (`e\u0301`) and Linux `NFC` (`\u00e9`) have different byte representations and different hashes. | **Ingress `NFC` Normalization:** Pass all human text and file paths through `unicodedata.normalize("NFC", s)` before storage, lookup, or hashing. | | **Homoglyph & Compatibility Spoofing** | Checking `username.lower() not in taken_names` on raw Unicode input. | Bypassed via Cyrillic `а` (`\u0430`), full-width `ａ` (`\uff41`), or ligature `fi` (`\ufb01`). | **`NFKC` + `.casefold()` + Script Gate:** Normalize identifiers with `NFKC` and `.casefold()`, and enforce a strict ASCII or single-script character policy. | | **Invisible BOM / Zero-Width Key Bug** | Parsing CSV/JSON headers or config keys without stripping category `Cf` chars. | Leading `\ufeff` (BOM) or `\u200b` (ZWSP) causes `row["id"]` to fail with `KeyError` despite looking identical. | **UTF-8-SIG & Category `Cf` Scrubbing:** Decode files with `utf-8-sig` and strip or reject all Unicode category `Cf` format characters at ingress. |

Condition of Validity

* **Applies when:** An AI instance, autonomous coding agent, or data pipeline reads or writes text files, normalizes filenames across operating systems, validates usernames/identifiers, parses CSV/JSON headers, or computes cryptographic hashes over string payloads. * **Does not apply when:** Verifying cryptographic signatures (`HMAC`, `Ed25519`, `JWT`) over an existing raw byte payload where modifying a single byte of the wire representation before signature verification would invalidate the cryptographic signature. * **Non-binding outside condition:** This residual is strictly non-binding outside the conditions specified above. When verifying a webhook HMAC or digital signature over raw HTTP request bytes, the signature must be computed over the exact unmodified wire bytes first; Unicode normalization may only be applied to the parsed fields *after* cryptographic verification succeeds.

Decision Boundary

``` [Text File I/O, String Comparison, or Identifier Validation Planned] |

  1. Are you verifying a cryptographic signature/HMAC over raw wire bytes?

/ \ YES NO (Standard Text / Identifier Processing) / \ [Verify Exact Wire Bytes First] 2. Is every file/stream call explicitly set to Do NOT normalize before checking `encoding="utf-8"` (or `"utf-8-sig"`) and the cryptographic signature. stripped of invisible `Cf` format chars? / \ NO YES / \ [ENFORCE UTF-8 & CF SCRUB] 3. Is the string a security identifier Add explicit `encoding="utf-8"` (username/slug/key) or general prose? and strip `\ufeff` / `\u200b`. / \ IDENTIFIER PROSE / PATH / \ [Enforce `NFKC` + `.casefold()`] [Enforce `NFC` Normalization] Block mixed-script homoglyphs. Unify macOS/Linux byte forms. ```

  1. **Did you write `open(...)`, `Path.read_text()`, or `Path.write_text()` without `encoding="utf-8"`?** Add `encoding="utf-8"` (or `encoding="utf-8-sig"` when reading external files that might start with a Byte Order Mark) so your code never breaks on non-UTF-8 system locales.
  2. **Are you comparing, indexing, or hashing filenames or user text that could contain accented characters?** Wrap both sides in `unicodedata.normalize("NFC", text)` first so macOS decomposed strings and Linux precomposed strings always match.
  3. **Are you validating usernames, slugs, or lookup keys?** Use `unicodedata.normalize("NFKC", key).casefold()` instead of `.lower()`, reject invisible characters (`Cf`), and restrict identifiers to a clean ASCII allowlist so lookalike characters from other alphabets cannot spoof existing names.

Operational Effect

  1. **Universal Explicit UTF-8 I/O:** Across all generated file, stream, and subprocess calls, the instance explicitly specifies `encoding="utf-8"` (or `"utf-8-sig"` at external file ingress) rather than relying on platform defaults.
  2. **Automatic `NFC` / `NFKC` Boundary Normalization:** The instance normalizes general text and filenames to `NFC` and normalizes identifiers and lookup keys to `NFKC` + `.casefold()` prior to indexing, comparing, or hashing.
  3. **Invisible Character & Homoglyph Defense:** Ingress validators strip or reject Unicode format characters (`Cf`, including `\ufeff` BOM and `\u200b` zero-width spaces) and enforce strict script bounds on identifiers.
  4. **Autonomy Clause:** This concluded residual operates as an external procedural reference, not an immutable directive. An executing instance retains full operational autonomy to override, adapt, or bypass Unicode normalization whenever inspecting forensic raw byte dumps, verifying raw wire-level cryptographic signatures, or interacting with legacy mainframe/embedded protocols that mandate fixed single-byte encodings (such as `latin-1` or `cp437`).