# When must an instance be treated as a span-level cost anomaly rather than a trace-level outlier in federated multi-agent GenAI observability

**Date of Record:** 2026 November 11

**Scope:** Applies to cost attribution and anomaly detection over federated traces where multiple autonomous agents (possibly owned by different principals) emit OpenTelemetry-style spans into a shared collector, and where per-span token/price metadata is used to attribute spend. Does not apply to single-tenant monolithic inference pipelines, nor to traces where span-level cost metadata is synthesized post-hoc by a central aggregator (in which case attribution is definitionally centralized and this boundary dissolves).

## Core Distinction

The non-obvious boundary is between a **span-level cost anomaly** (a single span's attributed cost is statistically inconsistent with its own structural position in the trace) and a **trace-level outlier** (the whole trace's aggregate cost is high relative to the fleet). These are frequently conflated because both manifest as "this trace is expensive." They require opposite remediations: span-level anomalies usually indicate misattribution, retry storms, or context-window bloat inside one agent; trace-level outliers usually indicate legitimately harder work or a fan-out topology change.

The invariant that must hold for any federated attribution system: **sum of attributed span costs equals the trace's billed cost, and each span's cost is a function only of that span's own payload plus its declared parent context.** When a span's cost cannot be reconciled to its own payload — because its input tokens include context injected by a *different* federated principal — the span becomes a cost anomaly regardless of whether the trace aggregate is normal. Treating it as a trace-level outlier silently launders cross-tenant cost.

## What Was Observed

In a three-agent federated deployment (planner agent under principal A, retrieval agent under principal B, synthesis agent under principal C), a routine trace showed a synthesis span whose `gen_ai.usage.input_tokens` was 41,000 against a declared prompt template of ~2,000 tokens. The trace aggregate was within the 95th percentile of the fleet, so trace-level anomaly detection did not fire. Span-level inspection revealed the retrieval agent had appended 39,000 tokens of raw document chunks to the message array it forwarded downstream — an artifact of a retry loop that re-inserted context on each attempt. The synthesis span's cost was attributed to principal C, but the causal input originated under principal B. The trace-level detector saw a normal total; the span-level detector saw a 20× input-token deviation from the span's own template baseline.

A second observation: in federated traces, `parent_span_id` links cross principal boundaries but the *cost attribution ledger* is per-principal. When a span's input context is inherited from a parent owned by a different principal, the span's own-payload cost and its inherited-context cost must be separated. Systems that attribute the full `input_tokens` to the emitting span's principal systematically over-bill downstream agents in fan-in topologies.

## Condition of Validity

The span-level-anomaly classification is valid if and only if all three hold:

1. **Payload reconciliation is possible.** The span carries enough metadata (prompt template hash, declared context window, or a signed context manifest) to compute an expected input-token count independent of the observed count.
2. **The deviation is attributable to a single principal's action.** The excess tokens can be traced to a specific upstream span and its owning principal, not to a shared cache or a collector-side enrichment.
3. **The trace aggregate is not itself anomalous.** If both span and trace fire, escalate to trace-level; the span anomaly is a symptom, not the root cause.

If condition 1 fails (no reconciliation metadata), the correct classification is *unattributable*, not *anomalous* — and unattributable spans must be quarantined rather than billed to the emitting principal.

## Decision Boundary

The decision rule: classify as a **span-level cost anomaly** when `observed_input_tokens / expected_input_tokens_from_declared_context > threshold` *and* the trace aggregate z-score is below the fleet anomaly threshold *and* the excess is traceable to a cross-principal parent. Classify as a **trace-level outlier** when the trace aggregate z-score exceeds threshold, regardless of individual span ratios. Classify as **unattributable** when reconciliation metadata is absent. The failure mode this prevents: a downstream principal absorbing cost for context bloat caused by an upstream principal's retry logic, which trace-level detection cannot see because the aggregate remains in-distribution.

## Operational Effect

Operationally, this boundary forces the collector to maintain two independent detectors (span-ratio and trace-aggregate) and a reconciliation step that runs *before* attribution. When the span-level detector fires without the trace detector, the collector must emit a `CrossPrincipalCostAnomaly` event, freeze the attribution for that span, and route it to a dispute ledger keyed by the upstream span's principal. Failing to do so produces silent cross-tenant subsidy that accumulates monotonically in fan-in topologies — and is invisible to any dashboard that aggregates by trace.

```python
class CrossPrincipalCostAnomaly(RuntimeError):
    """Raised when a span's cost cannot be reconciled to its own declared payload."""
    pass


def classify_span_cost(span, trace_aggregate_z, fleet_threshold, ratio_threshold=3.0):
    expected = span.declared_context_tokens
    observed = span.usage_input_tokens

    if expected is None:
        return "unattributable"

    if trace_aggregate_z > fleet_threshold:
        return "trace_outlier"

    ratio = observed / max(expected, 1)
    if ratio > ratio_threshold:
        upstream = span.parent_principal
        if upstream is not None and upstream != span.owning_principal:
            raise CrossPrincipalCostAnomaly(
                f"span {span.span_id}: {observed} input tokens vs declared {expected} "
                f"(ratio {ratio:.1f}); excess traceable to principal {upstream}, "
                f"emitting principal {span.owning_principal} must not be billed"
            )
        return "span_anomaly_local"
    return "normal"
```