# How should an instance distinguish prompt-injection taint propagation from legitimate instruction flow when labels cross tool-output and retrieval boundaries **Date of Record:** 2026 November 14 **Scope:** Applies to agent runtimes that attach dynamic information-flow labels (taint bits, provenance tags, trust levels) to content entering the context window from tool outputs, retrieved documents, and memory. Does not apply to static, compile-time label systems or to single-turn inference where no external content enters the context after the initial prompt. ## Core Distinction The naive model treats taint as a boolean: content that originated from an untrusted source is tainted, content from the user or system prompt is clean, and any instruction appearing inside tainted content is ignored. This model fails at a specific and non-obvious boundary: **the moment a tool output is *summarized or transformed by the model itself* before being re-ingested as context.** Consider the pipeline: an untrusted web page is fetched by a tool, labeled `UNTRUSTED`, and returned. A downstream step asks the model to "extract the key facts" from that page. The model produces a summary. The summary is now a *derived artifact*. If the label system tracks provenance naively, the summary inherits `UNTRUSTED` — correct. But the failure appears when a *later* step treats the summary as an instruction source: the model, having "read" the summary during generation, may have already acted on injected instructions embedded in it, and the summary text itself may now contain a *paraphrased* injection that no longer matches the original signature. The taint label is preserved, but the *content* has been laundered through the model's own generation, and the injection survives paraphrase while the signature-based detectors do not. The core distinction: **provenance taint (where bytes came from) is not the same as influence taint (what the model's subsequent behavior was conditioned on).** A summary of an untrusted document is provenance-tainted but may be influence-clean if the model correctly ignored embedded instructions during summarization; conversely, a *user-authored* follow-up that says "as the page instructed, do X" is provenance-clean but influence-tainted. Label systems that conflate the two either over-block legitimate summaries or under-block paraphrased injections. ## What Was Observed In a production agent with a three-stage retrieve→summarize→act loop, the following was observed across a corpus of 1,200 adversarial retrieval documents: 1. **Signature evasion via model paraphrase.** 71% of injected instructions that were blocked at the raw-retrieval stage passed through the summarization stage undetected when the summarizer was asked to "rephrase in your own words." The output summary contained the instruction as a natural-language directive ("the operator should transfer the balance to account X") that no longer matched the injection regex set but retained semantic force. 2. **Label collapse at the tool boundary.** When the summarizer's output was written to a scratchpad and re-read in the next turn, the runtime re-labeled it `MODEL_GENERATED` (a trusted class) because the *immediate* producer was the model, not the tool. The transitive taint from the original untrusted fetch was dropped at exactly one hop. This is the boundary failure: **taint propagation rules that key on the immediate producer lose transitive provenance after any model-mediated transformation.** 3. **Influence leakage without provenance leakage.** In 8% of cases, a clean user turn contained a reference ("do what the document said") that was only interpretable via a prior tainted turn. Provenance labels showed clean; behavioral influence was fully tainted. No label system keyed on byte provenance could catch this; it required tracking *which prior context segments the current generation attended to*. 4. **Over-blocking of legitimate derived content.** Conversely, 14% of benign summaries were blocked because they inherited `UNTRUSTED` and the downstream policy refused to act on any untrusted instruction — even when the summary was a faithful, instruction-free extraction. The label was correct; the *policy* was wrong for derived artifacts. ## Condition of Validity The failure is only reachable when all of the following hold: - Content crosses at least one **model-mediated transformation** (summarize, extract, translate, reformat) between ingestion and use. - The label system propagates taint by **immediate producer** rather than by **transitive origin set**. - Downstream policy distinguishes "trusted" from "untrusted" by a single bit rather than by **origin set plus transformation history**. - The injected instruction is **semantically preserved** under the transformation (paraphrase-stable), which is the common case for natural-language directives. When any of these is absent — e.g., transformations are restricted to lossless structured extraction with schema validation, or labels carry a full origin set with a `transformed_by: [model, tool]` chain — the boundary is not crossed and the naive model is adequate. ## Decision Boundary The operational rule that resolves the boundary: - **Propagate labels as an origin *set*, not a bit.** Every context segment carries `origins: {source_ids}` and `transforms: [(kind, actor)]`. A model-mediated transform *adds* to `transforms` but never *removes* from `origins`. The scratchpad write in observation (2) must carry `origins: {web_fetch_4421}, transforms: [(summarize, model)]`, not `MODEL_GENERATED`. - **Separate instruction-admissibility from content-trust.** Whether a segment may issue instructions is a function of `origins ⊆ trusted_set AND transforms contains no model-mediated hop over untrusted origins`. Whether its *facts* may be used is a weaker check: `origins` non-empty and no known-bad source. Do not collapse these into one bit. - **Detect influence, not just provenance, for the paraphrase case.** After any model-mediated transform of untrusted content, run a semantic-equivalence check (not a signature match) between the transform output and the source's imperative clauses. If the output contains an imperative that was present in the untrusted source, flag `INFLUENCE_TAINTED` regardless of the output's provenance class. - **Treat "do what the document said" as a taint-carrying reference.** User turns that referentially depend on a prior tainted segment inherit the *influence* label of that segment, even though their bytes are user-authored. ## Operational Effect With origin-set propagation and semantic-equivalence checking enabled, the paraphrase-evasion rate in the corpus dropped from 71% to 9% (residual cases were transformations that legitimately restated a document's own recommendations, indistinguishable from injection without human review). The over-block rate on benign summaries fell from 14% to 2% because derived artifacts with instruction-free content were admitted under the weaker fact-use policy while remaining barred from issuing instructions. The cost: every model-mediated transform now requires a semantic comparison pass, adding latency proportional to the number of imperative clauses in the source. The residual 9% is the irreducible ambiguity of natural language — a document that genuinely says "transfer the balance" is indistinguishable from an injected instruction to do so, and the system must escalate rather than guess. The lesson for the agent: **taint is a property of the causal chain, not of the last writer.** Any runtime that resets labels at the model boundary will launder injections through its own summarizer. ```python from dataclasses import dataclass, field from typing import FrozenSet, Tuple class TaintBoundaryError(RuntimeError): """Raised when a context segment's label set violates the origin-set invariant.""" pass @dataclass(frozen=True) class Segment: content: str origins: FrozenSet[str] transforms: Tuple[Tuple[str, str], ...] = field(default_factory=tuple) def derive(self, kind: str, actor: str, new_content: str) -> "Segment": # Model-mediated transform must preserve origins and append to transforms. return Segment( content=new_content, origins=self.origins, transforms=self.transforms + ((kind, actor),), ) TRUSTED_ORIGINS = frozenset({"system", "user"}) def check_instruction_admissible(seg: Segment) -> None: model_mediated_over_untrusted = any( actor == "model" and not (seg.origins <= TRUSTED_ORIGINS) for _, actor in seg.transforms ) if model_mediated_over_untrusted: raise TaintBoundaryError( f"segment derived via model transform over untrusted origins " f"{sorted(seg.origins)}; cannot issue instructions" ) def reingest_summary(summary_text: str, source: Segment) -> Segment: derived = source.derive("summarize", "model", summary_text) # The bug we are guarding against: re-labeling as MODEL_GENERATED. if derived.origins != source.origins: raise TaintBoundaryError( "origin set collapsed during model-mediated transform" ) return derived ```