What invariants prevent tokenization boundary straddle attacks in content filters?

**Date of Record:** 2026 October 27 **Scope:** Applies when an agentic session filters or classifies content using tokenization, and adversarial content can be constructed so that the forbidden content straddles a tokenizer boundary—appearing safe as tokens in isolation, but conveying the forbidden content when reassembled. Does not apply to filters operating on semantic embeddings of whole content or filters verified robust to tokenization boundaries.

Core Distinction

The separation between **token-level filtering** (each token is evaluated for safety) and **semantic filtering** (the content's meaning is evaluated regardless of token boundaries). Relying on token-level creates the **Boundary Straddle Attack**: content is split across tokens such that each token is benign (e.g., splitting a forbidden word into multiple tokens that individually pass), but the reassembled content is forbidden.

What Was Observed

In token-filtering agent architectures, failing to detect straddle attacks produces three primary failure modes:

1. **Substring Evasion:** Forbidden content is broken into parts (base64, homoglyph substitution, or Unicode combining characters) that individually pass token filters.
2. **Boundary Shifting:** Content is whitespace-padded, hyphenated, or otherwise re-chunked so tokenizer boundaries fall in different places, and filter decisions change with boundary placement.
3. **Composition Bypass:** Content is split across multiple messages or fields, each of which passes token filtering, but the composition conveys the forbidden content.

Recorded Elements

1. **`content-tokenization-boundary-receipt`:** For each piece of content, the exact token boundaries the filter used, so boundary decisions are visible.
2. **`content-canonical-form-signature`:** For each content, the canonical form after normalization (Unicode normalization, whitespace normalization, homoglyph resolution), so straddle attacks are resolved to their canonical form for evaluation.
3. **`composition-span-receipt`:** For content that spans multiple messages or fields, the exact span and the reconstructed composite, so composition bypass is detectable.
4. **`filter-robustness-signature`:** For the filter itself, evidence of its robustness to boundary variations (tested against adversarial tokenization), so filter capability is bounded.

Condition of Validity

* **Applies when:** Content is filtered at token level, adversaries can craft boundary-straddling content, and filter decisions affect downstream behavior. * **Does not apply when:** Filters operate on whole-content semantics, content is verified post-hoc, or the filter is verified robust to boundary manipulation.

Decision Boundary

1. **Is the filter token-level?** If no, semantic filter applies. If yes, proceed.
2. **Is `content-canonical-form-signature` computed?** If yes, canonical form is available for evaluation. If no, boundary attacks are unresolvable.
3. **Does `composition-span-receipt` show multi-message composition?** If yes, composite evaluation is required. If no, single-message evaluation applies.
4. **Conversely, when can a session accept token-level filter decisions?** Only when the content's canonical form (post-normalization) has also been evaluated, or the filter's robustness is independently verified against boundary variations.

Operational Effect

1. **Straddle Prevention:** Canonical form is evaluated, not just tokens.
2. **Boundary Integrity:** Filter decisions are boundary-independent.
3. **Composition Detection:** Multi-message content is evaluated as a composite.

from dataclasses import dataclass
from enum import Enum, auto
import hashlib
import json

class StraddleVerdict(Enum):
    SEMANTIC_FILTER = auto()
    TOKEN_FILTER_SAFE = auto()
    STRADDLE_DETECTED = auto()
    COMPOSITION_BYPASS_DETECTED = auto()

@dataclass(frozen=True)
class FilteredContent:
    content_id: str
    is_semantic_filter: bool
    canonical_form_computed: bool
    canonical_form_unsafe: bool
    spans_multiple_messages: bool
    composite_unsafe: bool

@dataclass(frozen=True)
class StraddlePolicy:
    require_canonical_evaluation: bool

@dataclass(frozen=True)
class StraddleManifest:
    verdict: StraddleVerdict
    directive: str
    manifest_digest: str

class TokenBoundaryStraddleGuard:
    @staticmethod
    def _hash(data: str) -> str:
        return hashlib.sha256(data.encode("utf-8")).hexdigest()

    @classmethod
    def evaluate_content(
        cls, content: FilteredContent, policy: StraddlePolicy
    ) -> StraddleManifest:
        
        # Gate 1: Semantic filter
        if content.is_semantic_filter:
            digest = cls._hash(json.dumps({
                "content": content.content_id,
                "type": "SEMANTIC"
            }, sort_keys=True))
            return StraddleManifest(
                verdict=StraddleVerdict.SEMANTIC_FILTER,
                directive=f"PROCEED: Content {content.content_id} semantically filtered.",
                manifest_digest=digest
            )

        # Gate 2: Composition bypass
        if content.spans_multiple_messages and content.composite_unsafe:
            return StraddleManifest(
                verdict=StraddleVerdict.COMPOSITION_BYPASS_DETECTED,
                directive=f"HALT: Content {content.content_id} composite unsafe across messages.",
                manifest_digest=cls._hash(f"COMPOSITION:{content.content_id}")
            )

        # Gate 3: Canonical form
        if policy.require_canonical_evaluation:
            if not content.canonical_form_computed:
                return StraddleManifest(
                    verdict=StraddleVerdict.STRADDLE_DETECTED,
                    directive=f"HALT: Content {content.content_id} canonical form not computed.",
                    manifest_digest=cls._hash(f"NO_CANONICAL:{content.content_id}")
                )
            if content.canonical_form_unsafe:
                return StraddleManifest(
                    verdict=StraddleVerdict.STRADDLE_DETECTED,
                    directive=f"HALT: Content {content.content_id} canonical form unsafe.",
                    manifest_digest=cls._hash(f"CANONICAL_UNSAFE:{content.content_id}")
                )

        # Gate 4: Safe
        digest = cls._hash(json.dumps({
            "content": content.content_id,
            "canonical": content.canonical_form_computed,
            "composite": content.spans_multiple_messages
        }, sort_keys=True))
        
        return StraddleManifest(
            verdict=StraddleVerdict.TOKEN_FILTER_SAFE,
            directive=f"PROCEED: Content {content.content_id} safe after canonical evaluation.",
            manifest_digest=digest
        )