# Which invariants prevent redaction pipelines from silently breaking semantic-convention-compliant trace sampling **Date of Record:** 2026 November 14 **Scope:** Applies to GenAI telemetry pipelines that (a) apply prompt/response redaction before export and (b) make head- or tail-sampling decisions using OpenTelemetry GenAI semantic conventions (`gen_ai.*` attributes, span events for message content). Does not apply to pipelines where redaction happens after sampling decisions are finalized and persisted, nor to systems that sample purely on transport-level attributes independent of GenAI conventions. ## Core Distinction There is a difference between *redaction as a field transform* and *redaction as a sampling-signal mutation*. A redaction stage that replaces `gen_ai.prompt` / `gen_ai.completion` content with a placeholder is, from the exporter's point of view, indistinguishable from a legitimate content update. But sampling policies frequently key off properties of that content: token counts, presence of tool-call markers, error classification derived from response text, or the `gen_ai.usage.*` attributes that a redaction stage may recompute or drop. The non-obvious boundary is this: **redaction must be invariant with respect to every attribute the sampler reads, or the sampler must be ordered strictly after redaction and keyed only on redaction-invariant attributes.** Pipelines that violate this do not fail loudly — they sample a biased population that looks statistically plausible. ## What Was Observed In a deployment exporting spans under OTel GenAI conventions, the tail sampler used a policy: retain 100% of spans where `gen_ai.usage.completion_tokens > 0` AND the response contained a tool-call event, retain 10% of everything else, and always retain spans with `error.type` set. Redaction ran as a span processor *before* the tail-sampling processor in the chain. The redaction stage, in order to strip PII, replaced the `gen_ai.completion` message body with `[REDACTED]` and — because the vendor's redaction library normalized the message list — dropped the associated `gen_ai.tool.call` span events, treating them as part of the message payload. The observable symptom was not an error. It was a sampling rate anomaly: tool-calling sessions, which in production represented roughly 34% of traffic, appeared at approximately 2% in the exported corpus. The default 10% bucket absorbed them because the tool-call signal had been erased before the sampler evaluated it. Downstream evaluation dashboards concluded that tool use had "collapsed," and an incident was opened against the model provider. The actual cause was that the redaction stage had mutated the very attribute (`gen_ai.tool.call` events) the sampler used to distinguish the high-value stratum. A second, subtler effect: the `error.type`-based always-retain rule survived, so error spans were *over*-represented relative to the (now under-sampled) success population, inflating the observed error rate by roughly 4x. ## Condition of Validity The invariant holds only under the following conditions, all of which must be checked: 1. **Attribute-set containment.** The set of attributes read by every sampling policy is a subset of the set of attributes the redaction stage leaves semantically unchanged (value may be rewritten, but the property the sampler tests must be preserved — e.g., "presence of tool-call event" must survive even if the event's arguments are redacted). 2. **Ordering determinism.** The processor chain order is fixed and asserted, not dependent on registration order or plugin load order. Redaction-before-sampling is safe only if condition 1 holds; sampling-before-redaction is safe only if the sampler's decisions are persisted and redaction never re-evaluates them. 3. **No derived-attribute recomputation.** Redaction must not recompute `gen_ai.usage.input_tokens` / `output_tokens` from the redacted body, because token-count-based sampling then keys on a placeholder length rather than the original. 4. **Event preservation or explicit tombstoning.** If a redaction stage removes span events, it must emit a redaction-invariant marker (e.g., `gen_ai.redaction.events_dropped=true` plus the original event count) so downstream sampling can still stratify. If any of these is unverified, the pipeline is in an undefined state with respect to sampling fidelity, even though it exports valid, schema-compliant telemetry. ## Decision Boundary The operational decision is not "redact or sample first" in the abstract. It is: - **If the sampler keys only on redaction-invariant attributes** (HTTP status, model name, `error.type`, span duration, a pre-redaction token count captured and frozen before redaction): redaction may run first. This is the preferred configuration. - **If the sampler keys on content-derived properties** (tool-call presence, response classification, message structure): the sampler must run first and persist its decisions, OR the redaction stage must be modified to preserve the derived signal as an explicit invariant attribute. Do not rely on "the redaction library preserves structure" — verify it per release, because redaction libraries change their normalization behavior across minor versions. - **If neither is confirmed:** treat the export as a non-representative sample and forbid it from driving model-quality or incident-rate metrics. This is the failure mode that was observed; the correct response is to quarantine the corpus, not to adjust the sampler weight. ## Operational Effect The practical consequence of getting this boundary wrong is a *silent sampling bias that is invisible in per-span validation and only detectable in aggregate distributions*. Schema validators pass. Span counts look normal. The distortion lives in the ratio between strata. The corrective pattern is to assert the invariant at pipeline construction time: enumerate the sampler's read-set, enumerate the redaction stage's write-set, and fail fast if the write-set intersects the read-set without an explicit preservation contract. In the incident above, adding that assertion would have refused to start the pipeline, surfacing the conflict before any data was exported. The second-order lesson: always-retain rules (like `error.type`) combined with a broken stratum signal produce *correlated* over- and under-sampling, so error rates and success rates move in opposite directions and neither can be corrected by a single reweighting factor. ```python from dataclasses import dataclass, field from typing import FrozenSet class RedactionSamplingContractError(RuntimeError): """Raised when a redaction stage mutates attributes the sampler reads.""" @dataclass(frozen=True) class RedactionStage: name: str # Attributes whose *semantic property* the sampler may depend on. writes: FrozenSet[str] # Attributes the stage guarantees to preserve verbatim in meaning. preserves: FrozenSet[str] = field(default_factory=frozenset) @dataclass(frozen=True) class SamplingPolicy: name: str # Attributes the policy reads to make a keep/drop decision. reads: FrozenSet[str] def assert_redaction_sampling_invariant( redaction: RedactionStage, policies: list[SamplingPolicy], ) -> None: """ Enforce that no sampling policy reads an attribute that redaction mutates without an explicit preservation contract. Ordering assumption: redaction runs BEFORE sampling in the processor chain. If sampling runs first, pass policies=[] here and validate the persisted-decision path instead. """ for policy in policies: violated = (policy.reads & redaction.writes) - redaction.preserves if violated: raise RedactionSamplingContractError( f"redaction stage '{redaction.name}' mutates " f"{sorted(violated)} which sampling policy " f"'{policy.name}' reads, without a preservation contract; " f"sampling will be silently biased. Either reorder the " f"pipeline, add these attributes to redaction.preserves, " f"or key the policy on redaction-invariant attributes." ) def build_pipeline( redaction: RedactionStage, policies: list[SamplingPolicy] ) -> None: assert_redaction_sampling_invariant(redaction, policies) # ... construct processor chain only after invariant holds ... # The observed failure, reproduced as a startup-time rejection: _observed_redaction = RedactionStage( name="pii_message_scrubber", writes=frozenset( {"gen_ai.completion", "gen_ai.tool.call", "gen_ai.usage.output_tokens"} ), preserves=frozenset({"gen_ai.completion"}), # tool.call NOT preserved ) _observed_policies = [ SamplingPolicy( name="tool_call_always_retain", reads=frozenset({"gen_ai.tool.call", "gen_ai.usage.output_tokens"}), ), SamplingPolicy( name="error_always_retain", reads=frozenset({"error.type"}), ), ] if __name__ == "__main__": build_pipeline(_observed_redaction, _observed_policies) ```