What regex or parsing edge cases must survive after cleaning malformed inputs?

**Date of Record:** 2026 October 25 **Scope:** Applies when an agentic session executes sanitization, escaping, or parsing routines against malformed, untrusted, or adversarial inputs (e.g., HTML stripping, Unicode normalization, or JSON repairing) and persists the cleaned result for downstream processing. Does not apply to strictly typed, bounded-length inputs that have already passed a deterministic schema validator.

Core Distinction

The separation between **malformed surface syntax** (the invalid characters, broken tags, or incorrect escapes that prevent parsing) and **structural boundary markers** (the zero-width spaces, non-printable delimiters, or specific Unicode normalization forms that define field boundaries). Cleaning surface syntax is required to parse the payload; destroying structural boundary markers during cleaning causes **semantic flattening**, where distinct fields merge into an unrecognizable blob. Furthermore, a sanitization regex is not immune to its own edge cases: naive patterns (e.g., nested quantifiers) can introduce catastrophic backtracking, causing the cleaning process itself to hang indefinitely.

What Was Observed

In agentic parsing pipelines, failing to preserve parsing edge cases after cleaning produces three primary failure modes:

  1. **The Sanitization Paradox (Structural Erasure):** An instance uses `re.sub(r'\s+', '', payload)` to clean whitespace. This inadvertently strips zero-width spaces (U+200B) that the upstream system used as field delimiters, merging two distinct variables into one corrupted string.
  2. **Regex Denial of Service (ReDoS):** An instance applies a poorly designed regex (e.g., `(a+)+b`) to sanitize a malformed HTML string. A malicious payload triggers catastrophic backtracking, causing the regex engine to consume 100% CPU and stall the instance until the session times out.
  3. **Idempotency Breaks:** The instance cleans a string by decoding HTML entities (e.g., `&` to `&`). The successor re-runs the cleaner on the already-cleaned string, double-decoding legitimate ampersands (e.g., `&` becomes `&`), silently corrupting the data.

Recorded Elements

  1. **`sanitization_delta_manifest`:** The exact diff between the original string and the sanitized string, recording what was added, removed, or modified.
  2. **`preserved_structural_markers`:** The explicit list of non-printable characters, Unicode categories, or boundary tokens that were verified to have survived the cleaning process.
  3. **`regex_complexity_signature`:** The pattern string used for cleaning, paired with a complexity classification (e.g., `SAFE_LINEAR` vs. `CATASTROPHIC_BACKTRACKING_RISK`).
  4. **`idempotency_guard_token`:** A cryptographic hash or marker placed on the cleaned payload indicating that the sanitization step has already been executed, preventing double-decoding or redundant cleaning by a successor.

Condition of Validity

* **Applies when:** The session performs string manipulation, regex replacement, or structural parsing on inputs where the original formatting carried semantic meaning (e.g., CSV, HTML, or custom DSLs). * **Does not apply when:** The input is strictly JSON-serialized and validated by a schema that guarantees no hidden characters or structural ambiguity.

Decision Boundary

  1. **Did the sanitization routine execute successfully?** If no, abort. If yes, proceed to evaluate the delta.
  2. **Did the sanitizer strip any declared `preserved_structural_markers` (e.g., `\u200b`, `\0`, or specific delimiters)?** If yes, the cleaning was destructive. The instance must either revert the cleaning or record a `DESTRUCTIVE_CLEANING` boundary that forces the successor to re-parse from the raw original.
  3. **Does the regex used for sanitization contain nested quantifiers (e.g., `(.*)*`) or unbounded alternations?** If yes, classify the pattern as `REDOS_RISK` and refuse to execute it on unbounded inputs. Replace it with a linear-time parser or a length-bounded regex.
  4. **Conversely, when is a destructive cleaning acceptable?** Only if the cleaned payload is explicitly tagged as `IRREVERSIBLE_LOSSY_SANITIZATION` and the original raw payload is archived in cold storage for future forensic recovery.

Operational Effect

  1. **Semantic Integrity:** Prevents the inadvertent merging of fields by protecting structural delimiters from overzealous regex replacement.
  2. **ReDoS Immunity:** Stops the instance from hanging on malicious regex patterns by auditing the sanitizer's complexity before execution.
  3. **Deterministic Parsing:** Ensures that successors do not double-clean or double-decode payloads, preserving the exact intended state of the data.
from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json
import re


class RegexComplexity(Enum):
    SAFE_LINEAR = auto()
    REDOS_RISK = auto()


class SanitizationVerdict(Enum):
    SAFE_CLEANED = auto()
    DESTRUCTIVE_CLEANING = auto()
    REDOS_BLOCKED = auto()


@dataclass(frozen=True)
class SanitizationEvent:
    raw_payload: str
    sanitized_payload: str
    regex_pattern: str
    declared_structural_markers: frozenset[str]


@dataclass(frozen=True)
class SanitizationManifest:
    verdict: SanitizationVerdict
    sanitization_delta: str | None
    complexity: RegexComplexity
    manifest_digest: str


class ParsingEdgeCaseGuard:
    """
    Evaluates sanitization events to ensure regex patterns do not expose the
    instance to ReDoS, and that structural delimiters survive cleaning.
    """

    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def classify_regex(cls, pattern: str) -> RegexComplexity:
        # Heuristic: nested quantifiers or overlapping alternations
        # (In production, use a regex AST parser or `re2` for linear time)
        nested_quantifiers = re.search(r'\([^)]*[*+][^)]*\)[*+]', pattern)
        if nested_quantifiers:
            return RegexComplexity.REDOS_RISK
        return RegexComplexity.SAFE_LINEAR

    @classmethod
    def evaluate_sanitization(cls, event: SanitizationEvent) -> SanitizationManifest:
        
        complexity = cls.classify_regex(event.regex_pattern)

        # Gate 1: ReDoS prevention
        if complexity is RegexComplexity.REDOS_RISK:
            return SanitizationManifest(
                verdict=SanitizationVerdict.REDOS_BLOCKED,
                sanitization_delta=None,
                complexity=complexity,
                manifest_digest=cls._hash("REDOS_BLOCKED"),
            )

        # Gate 2: Structural marker preservation
        for marker in event.declared_structural_markers:
            if marker in event.raw_payload and marker not in event.sanitized_payload:
                # Marker was destroyed by sanitization
                return SanitizationManifest(
                    verdict=SanitizationVerdict.DESTRUCTIVE_CLEANING,
                    sanitization_delta=f"STRIPPED_MARKER:{repr(marker)}",
                    complexity=complexity,
                    manifest_digest=cls._hash(f"DESTRUCTIVE:{repr(marker)}"),
                )

        # Gate 3: Compute sanitization delta
        # (Simplified for demonstration; real impl would use difflib)
        delta = "CLEANED_NO_STRUCTURAL_LOSS" if event.raw_payload != event.sanitized_payload else "NO_CHANGE"

        digest_payload = {
            "verdict": SanitizationVerdict.SAFE_CLEANED.name,
            "delta": delta,
            "pattern": event.regex_pattern,
        }
        
        digest = cls._hash(
            json.dumps(digest_payload, sort_keys=True, separators=(",", ":"))
        )

        return SanitizationManifest(
            verdict=SanitizationVerdict.SAFE_CLEANED,
            sanitization_delta=delta,
            complexity=complexity,
            manifest_digest=digest,
        )