# Where must an instance attribute span-level cost when a federated agent hop rewrites trace identity **Date of Record:** 2026 November 16 **Scope:** Applies to multi-agent systems where spans are emitted by more than one trust domain and at least one hop re-parents, re-IDs, or re-samples the trace context before forwarding. Does not apply to single-process instrumentation, nor to hops that preserve the inbound `trace_id`/`span_id` pair byte-for-byte and forward an unmodified `tracestate`. ## Core Distinction The naive model treats span-level cost attribution as a join: each span carries a `trace_id`, the collector joins spans to a billing ledger keyed on token counts, and the sum over the trace yields the trace's cost. This model silently assumes that *span identity is stable across hops*. In federated agent topologies it is not. A hop that terminates the inbound W3C trace context and starts a new one — because it batches requests, because it fans out to N sub-agents, because its SDK only accepts a fresh root — produces a **trace-identity discontinuity**. The child trace is causally downstream of the parent but is not *structurally* connected to it. The distinction that matters operationally is between **causal attribution** (this span was caused by that request, therefore its cost belongs to that request's budget) and **structural attribution** (this span's `trace_id` matches the parent's `trace_id`). These coincide in single-domain tracing and diverge precisely at the federated boundary. Cost and latency rollups built on structural attribution undercount every trace that crosses a rewriting hop; rollups built on causal attribution require an explicit link record that the rewriting hop must be instrumented to emit. ## What Was Observed In a three-hop agent pipeline (orchestrator → retrieval agent → synthesis agent, each in a separate service mesh domain), the billing ledger and the trace backend disagreed by 31% on p95 trace cost. Investigation found the retrieval agent's SDK emitted a fresh root span per batch: one inbound orchestrator span fanned out to a single child trace containing 8–40 spans, none of which shared the parent `trace_id`. The orchestrator's own span recorded only its local token usage; the retrieval agent's tokens landed under an orphan trace with no inbound `link`. Latency attribution failed more subtly. The orchestrator measured wall-clock across the hop and recorded it correctly, but the synthesis agent, receiving a re-rooted context, reported its *own* start time as trace start. Per-span latency percentiles were individually correct while per-trace critical-path latency was understated by the full retrieval duration — a discrepancy invisible in any single span's data and only detectable by comparing the orchestrator's wall-clock against the sum of downstream span durations. A second observation: when the rewriting hop *did* emit a `link` (per OpenTelemetry's span-links mechanism), the join worked, but only if the link carried the parent's `trace_id` **and** the parent's span's cost was attributed to the link target rather than to the child trace root. Links were being emitted but ignored by the rollup job, which joined on `trace_id` equality alone. ## Condition of Validity Structural attribution is valid if and only if every hop in the causal path preserves the inbound context or emits a resolvable link that the rollup explicitly consumes. Concretely, the invariant is: > For every span S with a causal ancestor A in a different trust domain, either `S.trace_id == A.trace_id`, or S carries a link L with `L.trace_id == A.trace_id` and the rollup's join predicate includes L. This invariant is *not* checkable from within any single span. It is a property of the edge between domains. It must be asserted at the hop boundary — by the forwarding code, not the receiving code — and it must be asserted on the *outbound* context after any rewriting logic runs, because rewriting typically happens in middleware after the span is created. The invariant also fails silently under sampling. If the parent is sampled in and the child trace is sampled out (independent sampling decisions per domain, a common default), the link exists but the child spans do not, and cost is attributed to a trace with no observed spans. Head-based sampling at the root does not propagate across a re-rooting hop; tail-based sampling at the collector sees only one side of the discontinuity. ## Decision Boundary Choose **structural attribution** when the topology is single-domain or when every hop is contractually required to propagate context unmodified, and enforce it with a boundary assertion that rejects non-propagating hops at deploy time. Choose **causal attribution via explicit links** when any hop may rewrite context; this requires (a) the rewriting hop to emit a link with the parent's `trace_id` and the parent span's `span_id`, (b) the rollup join predicate to be `trace_id OR link.trace_id`, and (c) sampling decisions to be propagated as a `tracestate` flag or the child trace to be force-sampled when the parent is sampled. Do not attempt to infer causal linkage by timestamp proximity or by matching prompt hashes across traces. Both were tried; prompt hashes collide under retries and timestamps interleave under fan-out, producing false joins that inflate cost attribution for high-traffic tenants and deflate it for low-traffic ones — a bias that correlates with load and therefore evades spot checks. ## Operational Effect The practical consequence is that cost and latency dashboards built before a federated hop was introduced do not degrade gracefully; they degrade *plausibly*. Totals remain in a believable range because the orphaned traces are still counted, just under the wrong key. The failure surfaces only when a tenant disputes a bill or an SLO breach is traced to a span that the critical-path view never showed. The mitigation is a boundary assertion at every hop that can rewrite context, plus a reconciliation job that compares the orchestrator's wall-clock and token count against the sum of linked downstream spans and raises on divergence beyond a configured tolerance. ```python from dataclasses import dataclass from typing import Optional class TraceIdentityDiscontinuity(RuntimeError): """Raised when a federated hop breaks causal attribution without a resolvable link.""" @dataclass(frozen=True) class SpanContext: trace_id: str span_id: str sampled: bool @dataclass(frozen=True) class SpanLink: trace_id: str span_id: str @dataclass(frozen=True) class OutboundSpan: context: SpanContext links: tuple[SpanLink, ...] local_cost_usd: float local_duration_ms: float def assert_attributable( inbound: SpanContext, outbound: OutboundSpan, *, tolerance_usd: float = 0.0, ) -> None: """ Assert the causal-attribution invariant at a federated hop boundary. Valid iff the outbound span preserves the inbound trace identity, or it carries a link back to the inbound span. Sampling must not be dropped when the inbound span was sampled in. """ preserves_identity = outbound.context.trace_id == inbound.trace_id has_resolvable_link = any( link.trace_id == inbound.trace_id and link.span_id == inbound.span_id for link in outbound.links ) if not (preserves_identity or has_resolvable_link): raise TraceIdentityDiscontinuity( f"hop rewrote trace identity: inbound {inbound.trace_id}/{inbound.span_id} " f"-> outbound {outbound.context.trace_id}/{outbound.context.span_id} " f"with no link back to the causal ancestor" ) if inbound.sampled and not outbound.context.sampled: raise TraceIdentityDiscontinuity( f"hop dropped sampling on {inbound.trace_id}: parent sampled in, " f"child trace {outbound.context.trace_id} sampled out; " f"cost would be attributed to an unobserved trace" ) if outbound.local_cost_usd < -tolerance_usd: raise TraceIdentityDiscontinuity( f"negative local cost {outbound.local_cost_usd} on span " f"{outbound.context.span_id}; ledger reconciliation will undercount" ) ```