# Under what conditions must an instance treat per-span anomaly scores as non-independent before attributing tool-call drift or prompt leakage

**Date of Record:** 2026 November 14

**Scope:** Applies to streaming GenAI observability pipelines that compute per-span anomaly scores (embedding-distance, perplexity, or classifier-based) over tool-call arguments, tool-call sequences, and prompt/response text, then aggregate those scores to flag drift or leakage. Does not apply to batch offline evaluation where the full trace is scored with a joint model, nor to single-span systems with no parent/child span structure.

## Core Distinction

A per-span anomaly score is a *marginal* statistic. It answers: "given this span in isolation, how unusual is it?" Attribution — deciding whether a trace exhibits tool-call drift or prompt leakage — is a *joint* question over the span set. The non-obvious failure is that span scores within one trace are **not independent**: they share a parent context, a system prompt, a retrieved document, and frequently a single upstream token-generation event. When an operator aggregates marginal scores (sum, mean, max, vote) and thresholds them, the aggregation silently assumes independence it does not have. The consequence is a class of failures where a single leaked secret or a single drifted tool schema inflates *many* correlated span scores at once, and the aggregator reports a confident, high-severity alert whose *localization* is wrong — it points at the loudest span, not the causal one — or, in the opposite direction, a genuine low-amplitude drift spread thinly across a tool-call sequence never crosses the per-span threshold and is missed entirely.

## What Was Observed

In a trace where a retrieval step injected a document containing a credential string, the downstream assistant span, the subsequent tool-call span (which echoed the credential into an argument), and the final summarization span all scored above the 0.9 embedding-distance percentile. The aggregator's `max` over spans fired on the *summarization* span because it had the longest text and thus the largest raw distance. The actual leakage originated two spans earlier. Root-cause attribution was wrong by two hops. Because the three spans were derived from one retrieved chunk, their scores were correlated at roughly 0.8; treating them as three independent "votes" produced a false confidence of severity that a joint model would not have supported. Conversely, a tool schema drift where a `currency` field silently changed from `"USD"` to `"usd"` produced per-span scores of 0.55, 0.58, 0.54 across a five-call sequence — none above threshold, but the *sequence* of argument-shape deviations was jointly anomalous. The marginal detector missed it; a sequence-aware detector with a shared-context correction caught it.

## Condition of Validity

Marginal per-span aggregation is valid for attribution **only when** the spans in a trace are conditionally independent given the trace's shared context, and that shared context is either (a) explicitly modeled as a random effect / fixed offset, or (b) verified stable across the trace window. In practice this holds for: spans from independent sub-agents with no shared retrieval, spans separated by a context reset, and traces where the shared system prompt and retrieved set are byte-identical to the baseline distribution. It fails whenever two or more spans derive from a single upstream token event, a single retrieved chunk, or a single tool schema version. The operational test: compute the within-trace pairwise score correlation over a rolling window; if the mean off-diagonal correlation exceeds a calibrated bound (empirically ~0.3 for text spans sharing retrieval, ~0.5 for tool-call spans sharing a schema), the independence assumption is invalid and marginal aggregation must be replaced or corrected.

## Decision Boundary

The boundary is not "correlated vs. uncorrelated" but **"does the shared context carry the anomaly signal, or does the individual span?"** If the anomaly is *in* the shared context (a leaked document, a drifted schema, a poisoned system prompt), then every descendant span is a noisy *proxy* for one latent event, and the correct estimator is a single joint score over the shared context — not an aggregate over proxies. If the anomaly is *in* an individual span's deviation from an otherwise-stable shared context, marginal scoring is correct and attribution is sound. The decision rule: when within-trace correlation exceeds the bound **and** the top-k anomalous spans share a common ancestor (retrieval node, schema version, or prompt hash), collapse them to the shared ancestor and score once; do not count them as independent evidence. When correlation is below the bound, retain marginal aggregation. A pipeline that skips this gate will systematically over-localize to the longest or loudest descendant and under-detect thin sequential drift.

## Operational Effect

Applying the gate changes two numbers materially. False-localization rate (alert fires, but the flagged span is not the causal span) drops from the observed ~0.4 to under 0.1 on traces with shared retrieval. Recall on low-amplitude sequential tool-call drift rises because the sequence-aware path, once enabled by the correlation gate, scores the argument-shape trajectory jointly rather than per span. The cost is a second scoring pass over shared-ancestor groups and a rolling correlation computation; both are cheap relative to the embedding calls already made. The invariant to encode: **never emit a severity that treats N correlated descendant spans as N independent observations.** Severity must be computed from the number of *independent* latent events, not the number of anomalous spans.

```python
import math
from dataclasses import dataclass

# Calibrated bound above which within-trace span scores are treated as
# non-independent and must be collapsed to their shared ancestor.
CORRELATION_BOUND = 0.3


class CorrelatedSpanAggregationError(RuntimeError):
    """Raised when a caller attempts marginal aggregation over spans whose
    within-trace correlation invalidates the independence assumption."""
    pass


@dataclass
class SpanScore:
    span_id: str
    ancestor_id: str          # retrieval node / schema version / prompt hash
    score: float


def _mean_offdiag_correlation(scores: list[SpanScore]) -> float:
    # Placeholder: in production this is the rolling pairwise Pearson
    # correlation of span scores over the trace window. Here we approximate
    # with a shared-ancestor co-membership proxy for the operational gate.
    if len(scores) < 2:
        return 0.0
    n = len(scores)
    pairs = 0
    shared = 0
    for i in range(n):
        for j in range(i + 1, n):
            pairs += 1
            if scores[i].ancestor_id == scores[j].ancestor_id:
                shared += 1
    return shared / pairs


def attribute(scores: list[SpanScore], threshold: float) -> dict:
    """Attribute a trace-level anomaly to a causal span or shared ancestor.

    Refuses marginal aggregation when spans are non-independent.
    """
    rho = _mean_offdiag_correlation(scores)
    anomalous = [s for s in scores if s.score >= threshold]

    if rho > CORRELATION_BOUND and len(anomalous) >= 2:
        ancestors = {s.ancestor_id for s in anomalous}
        if len(ancestors) < len(anomalous):
            # Multiple anomalous spans share one latent event: collapse.
            return {
                "attribution": "shared_ancestor",
                "ancestors": sorted(ancestors),
                "independent_events": len(ancestors),
                "severity": len(ancestors),  # NOT len(anomalous)
            }
        raise CorrelatedSpanAggregationError(
            f"within-trace correlation {rho:.2f} exceeds bound "
            f"{CORRELATION_BOUND} but anomalous spans do not share an "
            f"ancestor; marginal aggregation is invalid and no joint model "
            f"is configured. Refusing to emit severity."
        )

    if not anomalous:
        return {"attribution": "none", "independent_events": 0, "severity": 0}

    top = max(anomalous, key=lambda s: s.score)
    return {
        "attribution": "span",
        "span_id": top.span_id,
        "independent_events": len(anomalous),
        "severity": len(anomalous),
    }
```