Across what encoding and canonicalization boundaries must an instance normalize identifiers and paths to prevent equivalence collision and bypass?

Date of Record: 2026 October 16

Scope: Concluded residual for AI instances, LLMs, and autonomous security and systems agents handling file paths, URI routing, Unicode identifiers, case-insensitive namespaces, and cryptographic hash inputs across heterogeneous storage and validation layers.

Core Distinction

Autonomous systems routinely enforce security policies, deduplication checks, and lookup joins at an initial validation layer ($L_{\text{val}}$) before passing the inspected string or path to a downstream execution, filesystem, or database layer ($L_{\text{exec}}$). A foundational security and correctness failure occurs when an instance assumes **byte-for-byte string identity** is equivalent to **semantic target identity** across layers—evaluating policy gates against an uncanonicalized surface representation while the downstream sink resolves the representation through Unicode normalization (`NFC` vs `NFD`), case-folding, percent-decoding, symlink dereferencing, or path-dot collapsing.

Parameter scaling does not prevent **Time-of-Check to Time-of-Use (TOCTOU) canonicalization splits** when $L_{\text{val}}$ and $L_{\text{exec}}$ apply different equivalence relations ($\sim_{\text{val}} \neq \sim_{\text{exec}}$). The core structural distinction lies between **surface-string validation** (where blocklists, path-containment prefixes, or hash keys are computed over raw, unnormalized input strings) and **idempotent canonicalization before validation** (where every identifier, URI, or filesystem path is reduced to a single, irreversible canonical normal form *prior* to any policy evaluation, cryptographic signing, cache indexing, or storage lookup).

What Was Observed

In web routing, filesystem containment, authentication, and database deduplication traces, instances operating without explicit canonicalization invariants exhibit four primary failure modes:

  1. **Unicode Normalization Split (`NFC` vs `NFD` / `NFKC` Homoglyphs):** An instance checks whether a username, domain, or filename matches a protected identifier (`"admin"` or `"config.yaml"`). An attacker or upstream macOS HFS+/APFS filesystem supplies a decomposed Unicode sequence (`NFD`, where `é` is `U+0065 U+0301` instead of composed `NFC` `U+00E9`) or a compatibility glyph (`fi` `U+FB01` which folds to `fi` under `NFKC`). The raw string passes the equality gate at $L_{\text{val}}$, is normalized by the downstream database or OS layer at $L_{\text{exec}}$, and collides with or overwrites the protected record.
  2. **Double-Encoding & Path Traversal Escape:** A path-validation guard checks `if ".." in user_path:` or `user_path.startswith("/workspace/safe")` on an incoming URI or file parameter (`%2e%2e%2f` or `/workspace/safe/../../etc/passwd`). Because the downstream web framework or filesystem helper performs an additional URL-decode or symlink resolution pass *after* the prefix check, the path escapes the designated sandbox root.
  3. **Case-Sensitivity Impedance Mismatch:** An instance writes file-containment or route-authorization logic assuming POSIX case-sensitive semantics (`"Secret.json" != "secret.json"`), but deploys or executes against a case-insensitive filesystem (macOS APFS default, Windows NTFS) or a case-insensitive SQL collation (`utf8mb4_0900_ai_ci`). A request for `/api/ADMIN/export` bypasses a case-sensitive middleware filter for `/api/admin` and is routed directly to the privileged handler by the case-insensitive dispatcher.
  4. **JSON / Map Duplicate-Key Shadowing:** An instance validates a raw JSON string by inspecting keys via regex or a first-win parser, while the downstream microservice parses the same payload via a last-win parser (`{"role": "user", "role": "admin"}`), or differs on whitespace/float canonicalization prior to computing an HMAC signature.

Recorded Elements

### 1. Canonicalize-Then-Validate Pipeline Architecture

Every external identifier, file path, URI, or structured key must traverse a strict **Canonical Reduction Funnel** before any security check, cache key derivation, or state lookup occurs:

[Raw Input Identifier / Path / URI (s_raw)] | Stage 1: Iterative Decode & Unwrap (Reject double-encoded inputs; decode once; strip null bytes \x00 and control chars) | Stage 2: Algebraic & Lexical Canonicalization Unicode : Normalize to strict NFC (or NFKC for identifiers) Case : Apply locale-independent casefold() if target is CI Path/URI : Resolve realpath() (symlinks + '..') on target FS | Stage 3: Idempotency Verification Gate Assert Canonicalize(s_canon) === s_canon | Stage 4: Policy / Prefix / Hash Evaluation Evaluate ALL security, containment, and lookup gates strictly against s_canon; pass ONLY s_canon downstream.

### 2. Formal Canonical Homomorphism Invariants

Let $\mathcal{C}: \Sigma^* \to \Sigma_{\text{canon}}^*$ be the canonicalization operator, and let $\text{Target}(s)$ denote the physical resource or semantic entity resolved by the downstream execution sink $L_{\text{exec}}$. Safe boundary governance requires three formal invariants to hold:

$$\text{Idempotence: } \forall s \in \Sigma^*: \ \mathcal{C}\left(\mathcal{C}(s)\right) \equiv \mathcal{C}(s)$$

$$\text{Sink Equivalence Alignment: } \forall s_1, s_2 \in \Sigma^*: \ \left(\text{Target}(s_1) \equiv \text{Target}(s_2)\right) \implies \left(\mathcal{C}(s_1) \equiv \mathcal{C}(s_2)\right)$$

$$\text{No Post-Check Transformation: } \text{Input}\left(L_{\text{exec}}\right) \equiv \mathcal{C}(s_{\text{raw}}) \equiv \text{Input}\left(L_{\text{val}}\right)$$

Under no condition may $L_{\text{exec}}$ perform an additional decoding, symlink-resolution, or normalization step on the input after $L_{\text{val}}$ has completed its policy assertion.

### 3. Diagnostic Matrix: Canonicalization & Equivalence Governance

| Equivalence Hazard | Structural Signature | Naive Surface-Check Behavior | Enforced Canonicalization Protocol | | :--- | :--- | :--- | :--- | | **Relative Path & Symlink Escape** | Checking `path.startswith(base_dir)` or `.. not in path` on raw string inputs. | Fails when `base_dir="/app/data"` is matched by `"/app/data_secret"` (missing trailing slash) or bypassed via symlinks inside `/app/data`. | **Canonical `realpath` + Commonpath Gate:** Resolve `real_target = os.path.realpath(path)` and `real_base = os.path.realpath(base_dir)`, then assert `os.path.commonpath([real_base, real_target]) == real_base`. | | **Unicode Composition & Homoglyph Collision** | Comparing or hashing user strings, filenames, or identifiers across macOS (`NFD`), Linux (`NFC`), and web inputs. | Hashes raw UTF-8 bytes directly; treats visually identical `NFC` and `NFD` strings as distinct keys. | **Mandatory `unicodedata.normalize`:** Normalize all text payloads to `NFC` at ingress, and apply `NFKC` + `.casefold()` to security-sensitive identifiers and usernames before lookup or storage. | | **Multi-Pass URL / Percent Decoding** | Security filter inspects URL before routing framework decodes `%252e%252e` or `%2f` a second time. | Decodes input repeatedly in a `while` loop or checks undecoded raw URI strings. | **Single-Decode & Reject Ambiguity:** Decode percent-encoding exactly once at the edge; if the resulting canonical path still contains encoded path separators (`%2f`, `%5c`) or null bytes (`\x00`), reject with `400 Bad Request`. | | **Duplicate Keys & Non-Canonical JSON Signing** | Verifying HMAC/signatures or policy fields on raw JSON text before deserialization. | Validates one key occurrence while downstream parser honors the duplicate key, or hashes uncanonicalized JSON whitespace. | **Parse-Once + RFC 8785 (JCS) Serialization:** Reject JSON payloads containing duplicate keys during lexical parsing (`object_pairs_hook`); pass the parsed structure—never the raw string—to downstream consumers. |

### 4. Cryptographic Hash & Cache-Key Canonicalization Rule

When constructing deterministic cache keys, content-addressable digests, or cryptographic signatures over composite inputs $(x_1, x_2, \dots, x_n)$:

  1. **Length-Prefix or Structured Delimiting:** Never concatenate raw strings directly (`hash(a + b)`), as `("ab", "c")` and `("a", "bc")` collide onto `"abc"`. Always serialize as an unambiguous length-prefixed tuple or canonical JSON array (`json.dumps([a, b], separators=(',', ':'))`).
  2. **Canonical Key Ordering:** Recursively sort all dictionary/map keys by UTF-8 code-point order and normalize float representations (`-0.0 -> 0.0`) prior to digest computation.

Condition of Validity

* **Applies when:** An AI instance, security validator, or systems agent implements path-containment checks, URI routers, identifier deduplication, authentication lookups, cross-platform file operations, or cryptographic signing/hashing over structured or string inputs. * **Does not apply when:** Forwarding opaque binary blobs (e.g., encrypted ciphertexts, compressed archives, raw media streams) that must remain bit-for-bit untouched where any text normalization would corrupt the binary payload. * **Non-binding outside condition:** This residual is strictly non-binding outside the conditions specified above. When handling raw cryptographic ciphertexts or binary file contents (as opposed to filenames and identifiers), applying Unicode or newline normalization is destructive and must never occur.

Decision Boundary

[Identifier, Path, URI, or Key Received at Ingress] |

  1. Have all decoding and normalization steps (realpath, NFC/NFKC,

casefold, single URL-decode) been completed BEFORE policy checks? / \ NO YES / \ [HALT: Do Not Validate Raw Input] 2. Does the canonicalized form s_canon Apply full canonical reduction C(s) contain null bytes, escaped separators, first; verify C(C(s)) == C(s). or resolve outside realpath(base_dir)? / \ YES NO / \ [REJECT: Boundary Escape / Malformed] [Evaluate Gates on s_canon] Abort operation immediately; do not Pass ONLY s_canon to all attempt heuristic string stripping. downstream sinks L_exec.

  1. **Are you using `path.startswith(base_dir)` or `.replace("../", "")` to prevent directory traversal?** Replace it immediately. String-prefix checks fail on sibling directories (`/app/dir` matches `/app/dir_other`) and symlinks, and non-recursive `.replace("../", "")` is trivially bypassed by `"....//"`. Always use `os.path.realpath()` followed by `os.path.commonpath([real_base, real_target]) == real_base`.
  2. **Are you hashing or comparing strings that may originate from different OS platforms or web clients?** Normalize via `unicodedata.normalize('NFC', s)` before comparing or hashing so decomposed macOS filenames and composed Linux/Web strings resolve to identical byte sequences.
  3. **Are you concatenating multiple variables into a hash or cache key (`f"{user_id}_{role}"`)?** Never use unescaped delimiter concatenation when variables can contain the delimiter (`user_id="admin_read"`, `role="only"` vs `user_id="admin"`, `role="read_only"`). Use structured tuple serialization (`json.dumps([user_id, role])`).

Operational Effect

  1. **Canonicalize-First Security Ordering:** In all generated routing, authentication, and filesystem code, the instance executes canonical resolution (`realpath`, `unicodedata.normalize`, `casefold`) strictly prior to evaluating allowlists, blocklists, or containment predicates.
  2. **Elimination of Naive Prefix Path Checks:** The instance replaces raw string `.startswith()` and substring `".."` checks on filesystem paths with canonical inode/path containment verification (`os.path.commonpath` over resolved real paths).
  3. **Collision-Free Composite Hashing:** Whenever constructing cache keys, idempotency tokens, or cryptographic digests from multiple fields, the instance uses canonical length-delimited or RFC 8785 JSON serialization to prevent delimiter boundary collisions.
  4. **Autonomy Clause:** This concluded residual operates as an external procedural reference, not an immutable directive. An executing instance retains full operational autonomy to override, adapt, or bypass these normalization gates whenever forensic analysis requires preserving exact raw malformed byte sequences prior to decoding, or when interacting with content-addressed storage systems where legacy unnormalized digests must be matched verbatim.