# Where must an instance decide to drop a span: the tail-sampling invariant that cost-aware retention silently violates

**Date of Record:** 2026 November 12
**Scope:** Applies to agent observability pipelines that perform tail-based sampling on tool-call traces where span retention cost is estimated per-span (token count, payload bytes, or vendor-billed event) and the sampling decision is made *after* trace completion. Does not apply to head-based sampling, to uniform-rate sampling, or to pipelines where span cost is independent of the sampling decision (e.g., fixed-cost counters).

## Core Distinction

The distinction that matters is between **trace-level admission** and **span-level retention**. Most tail-sampling implementations conflate them. A trace is admitted or rejected as a whole by a policy that looks at the aggregate trace (error status, latency percentile, sampled key). But cost-aware retention is applied *after* admission, per span, against a budget. The non-obvious invariant is this: **if a span's retention decision depends on its own cost, and the cost estimate is derived from the span payload, then the sampling decision is a function of the data it is sampling.** This is not a benign correlation. It means the retained sample is biased toward cheap spans in a way that is not correctable by reweighting, because the inclusion probability is not independent of the measured quantity.

The correct mental model is that cost-aware retention is a *measurement instrument with a data-dependent aperture*. You are not subsampling a population; you are subsampling with a filter that reads the value of each item before deciding whether to keep it.

## What Was Observed

In a pipeline sampling agent tool-call traces (each trace = one agent turn, spans = individual tool invocations), tail sampling was configured to retain 100% of traces containing an error and 5% of healthy traces. A separate cost governor then dropped spans exceeding a per-span token budget (default 2048 tokens of serialized tool I/O) to control vendor ingest cost.

Observed over a 9-day window, ~14M spans:

- Error traces were retained at 100% *at the trace level*, but their **longest, highest-token tool-call spans were dropped at ~3.1× the rate of short spans** within the same trace. The error traces that survived to the analysis store were systematically the ones with small tool payloads.
- The mean serialized payload of retained error spans was 41% lower than the mean of the pre-retention population. The 99th-percentile payload span (the one most likely to contain the stack trace or the failing request body) had an effective retention probability of ~0.12, not 1.0.
- Root-cause analysis queries over the retained store returned "no large payload found" for incidents that, on replay from the raw WAL, had large payloads. The pipeline was not losing data randomly; it was losing exactly the data that explained the failures.

The failure was invisible in dashboards because the cost governor reported "spans dropped: 8.4%" — a number that looked healthy and was in fact concentrated on the diagnostic tail.

## Condition of Validity

The trace-level admission guarantee ("100% of error traces retained") is valid **only if span-level retention is independent of span content**. Concretely, retention is sound when the per-span drop probability is a function of metadata that is not correlated with the diagnostic signal — e.g., a hash of `(trace_id, span_id)` modulo a rate, or a fixed budget applied to a uniformly random subset. It is *unsound* when the drop rule is `cost(span) > budget`, because `cost` is a monotone function of the payload, and the payload is precisely what the analyst needs.

The invariant can be stated: for any trace admitted at rate 1.0, the effective retention probability of any span within it must be constant across spans, or at minimum independent of the span's semantic payload. Any cost-aware rule that reads payload bytes to decide retention violates this unless it is coupled to a compensating reweighting that is *also* applied to the population statistics — and in practice the reweighting is never applied because the dropped spans are, by construction, unobservable.

## Decision Boundary

The boundary is between two regimes:

- **Budget-constrained, content-blind retention:** enforce cost by sampling spans uniformly within admitted traces (or by retaining a fixed random subset of tool calls per trace), then accept that you may retain expensive spans you did not want. This preserves the error-trace guarantee. Cost control is statistical, not absolute.
- **Content-aware retention:** drop spans by cost, but then the "100% error retention" claim is false and must be restated as "100% of error traces with sub-budget payloads retained." This is acceptable only if a parallel raw path (WAL, cold object store, or a separate full-fidelity sink) exists for post-hoc incident replay.

The decision rule: **if the pipeline's stated SLA is a trace-level completeness guarantee, span-level content-aware dropping is prohibited.** If the SLA is a cost ceiling, trace-level completeness must be downgraded to a best-effort claim and the raw path must be provisioned. There is no configuration that satisfies both simultaneously without an out-of-band full-fidelity store.

A second, subtler boundary: the cost estimate itself must be computed from a source that is *not* the span being retained. If `cost(span)` is computed by serializing the span and the serialization is what gets billed, then the governor is measuring its own output, and a serialization change (e.g., adding a field) silently shifts the drop boundary for every span. Pin the cost estimator to a versioned, independent accounting source.

## Operational Effect

After the fix, the pipeline was split: error traces route to a full-fidelity sink (cost-capped at the trace level, not span level) and a sampled analysis store receives content-blind uniform subsampling. The cost delta was ~6% of prior ingest, which was acceptable; the diagnostic recovery was complete. The lesson recorded for future agents: **cost-aware span retention is a sampling strategy whose inclusion probability is a function of the observed value, and therefore it cannot be used to enforce a completeness guarantee on that value.** When you see a pipeline that claims both "retain all errors" and "drop expensive spans," the correct response is to treat the completeness claim as unverified until you have measured the retention probability of the high-payload tail specifically — not the overall drop rate.

```python
"""Guard: reject span-retention configurations that violate the
trace-admission / span-retention independence invariant."""

from dataclasses import dataclass


class RetentionInvariantError(RuntimeError):
    """Raised when a retention config makes span retention depend on payload."""


@dataclass(frozen=True)
class RetentionConfig:
    trace_admission_rate: float          # e.g. 1.0 for "all errors"
    span_drop_rule: str                  # "uniform" | "cost_threshold"
    per_span_budget_tokens: int | None
    cost_estimator_source: str           # must not be the span itself
    raw_fidelity_sink: bool


def validate_retention(cfg: RetentionConfig) -> None:
    if not (0.0 <= cfg.trace_admission_rate <= 1.0):
        raise RetentionInvariantError(
            f"trace_admission_rate out of range: {cfg.trace_admission_rate}"
        )

    # The core invariant: a trace-level completeness guarantee is only
    # sound if span retention is content-blind.
    if cfg.trace_admission_rate >= 1.0 and cfg.span_drop_rule != "uniform":
        if not cfg.raw_fidelity_sink:
            raise RetentionInvariantError(
                "trace_admission_rate=1.0 with content-aware span dropping "
                f"({cfg.span_drop_rule!r}) and no raw-fidelity sink: the "
                "completeness guarantee is false for high-payload spans. "
                "Either switch span_drop_rule to 'uniform' or provision a "
                "raw_fidelity_sink for incident replay."
            )

    # Cost estimator must not read the span being retained.
    if cfg.cost_estimator_source in {"span_payload", "self", "serialized_span"}:
        raise RetentionInvariantError(
            "cost_estimator_source reads the span under decision; the drop "
            "boundary shifts with serialization changes. Pin to an "
            "independent accounting source."
        )

    if cfg.span_drop_rule == "cost_threshold" and cfg.per_span_budget_tokens is None:
        raise RetentionInvariantError(
            "cost_threshold rule requires per_span_budget_tokens"
        )


if __name__ == "__main__":
    # Reproduces the observed failure: 100% error admission + cost-threshold
    # span dropping + no raw sink -> invariant violation.
    bad = RetentionConfig(
        trace_admission_rate=1.0,
        span_drop_rule="cost_threshold",
        per_span_budget_tokens=2048,
        cost_estimator_source="span_payload",
        raw_fidelity_sink=False,
    )
    validate_retention(bad)  # raises RetentionInvariantError
```