# What happens when an instance attributes token cost to a span before its parent trace context is federated

**Date of Record:** 2026 November 13

**Scope:** Applies to multi-agent systems where spans are emitted by independently-deployed agents and joined into a single trace by a federation layer (e.g., a collector or trace-store merge) that resolves cross-agent parent references asynchronously. Does not apply to single-process instrumentation where parent-child relationships are established synchronously in memory before span emission, nor to systems where cost is attributed only at trace-close time by a single authoritative writer.

## Core Distinction

The distinction that matters is between *span-local cost* (tokens consumed by a single LLM call, attributable to the emitting agent the moment the call returns) and *trace-attributed cost* (the share of a trace's total cost that rolls up to a given parent agent or workflow node). Span-local cost is knowable synchronously. Trace-attributed cost is not, because the parent reference of a cross-agent span may point to a span that has not yet arrived at the federation layer.

The non-obvious failure is this: an agent that computes and emits *rolled-up* cost at span-emission time (rather than deferring rollup to trace-close) will silently mis-attribute cost whenever the parent span is federated after the child. The child's cost is attributed to the child's local subtree, then the parent arrives, then the parent's own rollup double-counts or drops the child depending on the merge strategy. There is no error; the numbers are internally consistent and wrong.

## What Was Observed

In a three-agent pipeline (planner → retriever → synthesizer) where each agent runs in a separate deployment and emits OTLP spans to a shared collector, the synthesizer's spans routinely arrived before the planner's parent span. The federation layer buffered orphaned spans for a fixed grace window (default 30s) and then, on timeout, re-parented them to a synthetic root. Under normal load the planner's span arrived within the window and the join succeeded. Under planner cold-start (model load, ~40s), the synthesizer's spans timed out and were re-parented to the synthetic root.

The cost attribution service consumed the joined trace and reported per-agent cost. In the timeout case, the synthesizer's token cost was attributed to the synthetic root, not to the planner's workflow node. Downstream chargeback showed the synthesizer billed for its own tokens but the planner's workflow node showed zero downstream cost — the planner appeared to consume only its own prompt tokens. Aggregate trace cost was correct (the synthetic root absorbed the orphan), but the per-node breakdown was wrong in a way that only appeared when the planner was slow to start.

The invariant that was violated: *for every trace, the sum of cost attributed to non-root nodes plus the synthetic-root cost must equal total trace cost, AND no non-root node's cost may be attributed to the synthetic root.* The first clause held. The second did not.

## Condition of Validity

The rollup-at-emission approach is valid only if, for every cross-agent edge, the parent span is guaranteed to be present at the federation layer before the child span is processed. In practice this holds when:

- Parent and child are emitted by the same process (synchronous parent context), or
- The federation layer enforces causal ordering (child spans are held until parent arrival with no timeout, or the timeout is longer than the maximum parent-emission latency including cold-start), or
- Cost rollup is deferred to trace-close, where the full span set is known.

It does **not** hold whenever the federation grace window can be exceeded by any agent's cold-start or queueing latency, which in federated multi-agent systems is the common case, not the exception. The validity condition is therefore a *latency bound on the slowest parent emitter*, and that bound is not something the child agent can observe or enforce.

## Decision Boundary

The decision is between two strategies, and the boundary is whether the federation layer can guarantee parent-before-child processing:

- **If the federation layer guarantees causal ordering** (no timeout-based re-parenting, or timeout strictly greater than worst-case parent latency): roll up cost at emission. Cost is available in real time and per-node attribution is correct.
- **If it does not** (timeout-based re-parenting, best-effort join): do not roll up at emission. Emit *span-local* cost only, tag orphaned spans explicitly with a `federation_orphan` marker, and perform trace-attributed rollup at trace-close against the final joined span graph. Accept that per-node cost is unavailable until trace close.

The failure mode to guard against is a system that *appears* to be in the first regime (joins usually succeed) but is actually in the second (joins fail under cold-start). The boundary is not "does it work most of the time" but "is the timeout provably above the maximum parent-emission latency." If that proof cannot be produced, treat it as the second regime.

## Operational Effect

When the boundary is crossed silently, the observable symptoms are: (1) per-node cost breakdowns that sum correctly at trace level but assign cost to the synthetic root; (2) chargeback reports where a workflow node's downstream cost is zero despite downstream agents consuming tokens; (3) the discrepancy appearing only under load or cold-start, so it is invisible in steady-state testing. The remediation is to detect orphan re-parenting at the federation layer and either (a) fail the trace's cost rollup loudly rather than absorb the orphan into the synthetic root, or (b) defer rollup to trace-close. The following guard raises rather than silently mis-attribute:

```python
class CostAttributionError(RuntimeError):
    """Raised when trace cost cannot be attributed to a real parent node."""
    pass


def attribute_cost(span_graph: dict, span_local_cost: dict) -> dict:
    """
    span_graph: {span_id: {"parent_id": str | None, "orphaned": bool}}
    span_local_cost: {span_id: float}
    Returns {node_id: rolled_up_cost} or raises CostAttributionError.
    """
    synthetic_root = "__synthetic_root__"
    attributed: dict[str, float] = {}

    for span_id, meta in span_graph.items():
        cost = span_local_cost.get(span_id, 0.0)

        if meta["orphaned"]:
            # A span was re-parented to the synthetic root because its real
            # parent missed the federation grace window. Rolling this up
            # silently would mis-attribute cost to the synthetic root.
            raise CostAttributionError(
                f"span {span_id} was re-parented to {synthetic_root} "
                f"(parent missed federation window); refusing to roll up "
                f"{cost} tokens to synthetic root — defer to trace-close or "
                f"extend the federation grace window beyond max parent latency"
            )

        parent = meta["parent_id"]
        if parent is None:
            attributed[span_id] = attributed.get(span_id, 0.0) + cost
        else:
            if parent not in span_graph:
                raise CostAttributionError(
                    f"span {span_id} references parent {parent} not present "
                    f"in joined graph; rollup is unsound"
                )
            attributed[parent] = attributed.get(parent, 0.0) + cost

    return attributed
```