# How does an instance detect the orphaned span boundary in GenAI observability cost attribution when per-agent token accounting diverges from per-tenant rollups **Date of Record:** 2026 November 15 **Scope:** Applies to multi-tenant GenAI observability pipelines that emit per-agent, per-tool, and per-tenant token/latency spans and reconcile them into a billing rollup. Does not apply to single-tenant deployments or to pipelines where billing is computed directly from provider invoices rather than from internally emitted spans. ## Core Distinction The failure is not "missing spans" or "sampling loss" — those are detectable by comparing emitted span counts against request counts. The non-obvious boundary is the **orphaned span**: a span that was correctly emitted, correctly attributed to an agent and a tool, and correctly counted in the per-agent and per-tool aggregation, but whose `tenant_id` was inherited from a parent context that had already been closed or replaced before the span was flushed. The span is present in the per-agent ledger and absent from the per-tenant ledger. Sum-of-agents exceeds sum-of-tenants by a delta that is *not* constant, *not* proportional to traffic, and *not* zero-mean — it drifts with concurrency. The distinction that matters operationally: a lost span produces a *negative* residual (tenant total < agent total by exactly the missing tokens). An orphaned span produces a *positive* residual that is masked because the tenant rollup is computed from a different aggregation path (typically a streaming group-by keyed on `tenant_id`) than the agent rollup (typically a batch reduce keyed on `agent_id`). The two paths never cross-check each other unless an explicit reconciliation job exists. ## What Was Observed In a pipeline processing ~4.2M spans/hour across 1,900 tenants, a reconciliation job comparing `sum(per_agent.tokens)` against `sum(per_tenant.tokens)` per 5-minute window reported a residual that oscillated between 0.3% and 2.1% of total tokens. The residual correlated with `p99` concurrent agent invocations (`r=0.81`) but *not* with error rate, *not* with retry count, and *not* with any single tenant's traffic share. Root cause isolated to a context-propagation race: the agent runtime used a `contextvars`-based tenant scope that was reset in a `finally` block. When a tool call fanned out to a subagent and the parent scope was reset before the subagent's span exporter flushed (async, batched, 250ms flush interval), the subagent span read `tenant_id=None` from the context. The exporter's default behavior on `tenant_id=None` was to emit the span with `tenant_id="unattributed"` — a valid string, a valid span, a valid per-agent contribution, and a silent per-tenant loss. The per-agent rollup keyed on `agent_id` and never inspected `tenant_id`, so it counted the tokens. The per-tenant rollup keyed on `tenant_id` and bucketed them into a synthetic tenant that no invoice ever read. The tell that made this detectable only after the fact: the `unattributed` bucket had a token count that was *stable under load* but *unstable in membership* — the same tokens appeared under different agent IDs across windows because the subagents rotated. A per-agent view showed no anomaly. A per-tenant view showed a small synthetic tenant with implausibly high per-agent cardinality. ## Condition of Validity This boundary condition holds only when all of the following are true, and the detection logic must be gated on them: 1. Per-agent and per-tenant rollups are computed by **independent aggregation paths** (different keys, different execution models). If both are derived from the same group-by, the residual is structurally zero and the check is vacuous. 2. Span export is **asynchronous and batched**, with a flush interval longer than the minimum parent-scope lifetime. Synchronous export eliminates the race. 3. The context propagation mechanism uses **mutable scope reset** (e.g., `contextvars.reset`, thread-local pop) rather than immutable inheritance. Immutable propagation (passing a frozen context object down the call chain) eliminates the race. 4. There exists a **synthetic or default tenant bucket** that accepts `tenant_id=None` or `tenant_id=""` without rejecting the span. If the exporter drops unattributed spans, the failure becomes a detectable negative residual instead of a silent positive one. If any of these is false, the orphaned-span boundary does not apply and the reconciliation residual has a different explanation (sampling, clock skew, or genuine loss). ## Decision Boundary The operational decision is: **when the per-agent vs. per-tenant residual is positive and exceeds a threshold, do not page on-call for "lost tokens" — page for "orphaned attribution" and route to the context-propagation owner, not the billing owner.** The threshold must be expressed as a *ratio of the synthetic-tenant bucket to total tokens*, not as an absolute residual, because the residual's magnitude is a function of concurrency and the synthetic bucket's magnitude is the actual loss. A residual of 2% with a synthetic bucket of 0.01% indicates a different bug (double-counting in the agent path). A residual of 0.3% with a synthetic bucket of 0.3% indicates orphaned spans. The decision to *block* a billing run must be gated on the synthetic bucket exceeding a contractual tolerance, not on the residual. Blocking on the residual produces false positives from benign aggregation-order differences; blocking on the synthetic bucket catches only true attribution loss. ## Operational Effect The practical consequence of missing this boundary is a billing system that under-charges tenants in proportion to their subagent fan-out — the heaviest users of multi-agent orchestration are systematically under-billed, and the under-billing is invisible because the per-agent dashboards (which engineering watches) are correct. Finance sees a revenue shortfall with no corresponding anomaly in the engineering observability stack. The fix is not in the billing code; it is in the span exporter's handling of a null tenant context, plus a reconciliation gate that treats the synthetic bucket as a first-class SLO. The secondary effect: once the synthetic bucket is monitored, its *cardinality* (distinct agent IDs contributing) becomes a leading indicator of context-propagation regressions, detectable hours before the token residual crosses any threshold. ```python from __future__ import annotations from dataclasses import dataclass class OrphanedAttributionError(RuntimeError): """Raised when per-agent token totals exceed per-tenant totals via a synthetic bucket.""" @dataclass(frozen=True) class RollupWindow: window_id: str per_agent_tokens: int per_tenant_tokens: int synthetic_tenant_tokens: int synthetic_tenant_agent_cardinality: int total_tokens: int def reconcile_attribution( window: RollupWindow, synthetic_ratio_tolerance: float = 0.001, cardinality_tolerance: int = 25, ) -> None: if window.total_tokens <= 0: raise OrphanedAttributionError( f"window {window.window_id} has non-positive total_tokens={window.total_tokens}" ) residual = window.per_agent_tokens - window.per_tenant_tokens synthetic_ratio = window.synthetic_tenant_tokens / window.total_tokens # Benign: no synthetic bucket growth, residual within aggregation-order noise. if synthetic_ratio <= synthetic_ratio_tolerance and residual <= 0: return # Benign: negative residual is genuine loss, not orphaned attribution. if residual < 0 and synthetic_ratio <= synthetic_ratio_tolerance: return if ( residual > 0 and synthetic_ratio > synthetic_ratio_tolerance and window.synthetic_tenant_agent_cardinality > cardinality_tolerance ): raise OrphanedAttributionError( "orphaned span boundary crossed in window " f"{window.window_id}: residual={residual} tokens, " f"synthetic_ratio={synthetic_ratio:.4%} (tol {synthetic_ratio_tolerance:.2%}), " f"synthetic_agent_cardinality={window.synthetic_tenant_agent_cardinality} " f"(tol {cardinality_tolerance}). Suspect mutable-context reset before " "async span flush; route to context-propagation owner, not billing." ) # Residual positive but synthetic bucket small: aggregation-order or double-count. if residual > 0 and synthetic_ratio <= synthetic_ratio_tolerance: raise OrphanedAttributionError( f"window {window.window_id}: positive residual={residual} without synthetic " "bucket growth — suspect agent-path double-count, not orphaned spans." ) def run_reconciliation(windows: list[RollupWindow]) -> list[str]: failures: list[str] = [] for w in windows: try: reconcile_attribution(w) except OrphanedAttributionError as exc: failures.append(str(exc)) return failures ```