# Under what conditions must an instance treat a trace_id as a correlation key rather than an identity **Date of Record:** 2026 November 15 **Scope:** Applies to multi-agent systems where LLM spans and tool/retrieval spans are emitted into a shared trace store and joined downstream by `trace_id` and `span_id`. Does not apply to single-process synchronous LLM calls, or to systems where the trace store is the sole writer of span identifiers (i.e., no client-supplied IDs). ## Core Distinction A `trace_id` is a *correlation key*: it asserts that two spans belong to the same logical request. It is not an *identity*: it does not assert that the spans were produced by the same process, the same agent instance, or even the same execution. In distributed agent traces, the same `trace_id` is routinely propagated across process boundaries, retries, and re-entrant tool calls, so any invariant of the form "all spans with trace_id X share property P" is only as strong as the propagation discipline that produced X. The non-obvious failure: a retried child invocation that reuses the parent `trace_id` but receives a *new* `span_id` produces a trace whose span tree is a DAG with duplicated semantic nodes, not a tree. Correlation joins on `trace_id` alone will silently double-count tokens, costs, and latency percentiles because the retried span and the original span are both present with the same semantic role but distinct identifiers. ## What Was Observed In a three-agent pipeline (planner → retriever → synthesizer), a transient 503 from the retriever's vector store caused the orchestrator to retry the retriever call. The retriever emitted its first span with `span_id = S1`, failed, and the retry emitted `span_id = S2`. Both carried the parent `trace_id = T` and the same `parent_span_id` (the planner's span). Downstream aggregation computed: - **Token totals:** summed over all spans in `T`, yielding 1.87× the true billed token count because the failed attempt's partial prompt tokens were counted alongside the successful retry. - **Latency p50:** computed over span durations grouped by `trace_id`, which mixed the failed attempt (fast, 40ms) with the successful retry (slow, 2.1s), producing a p50 that matched neither. - **Agent handoff graph:** the retriever node appeared twice in the DAG, inflating its in-degree and causing a cycle-detection heuristic to flag the trace as malformed. The root cause was not the retry itself; it was the assumption that `(trace_id, semantic_role)` uniquely identifies a span. It does not. Only `(trace_id, span_id)` does, and even then only if `span_id` generation is collision-free across retry attempts. ## Condition of Validity The correlation-key treatment of `trace_id` is valid *only* when all of the following hold: 1. Every span writer generates `span_id` from a source that is monotonic or random-with-sufficient-entropy per attempt (not per logical call). 2. Retries are represented as *sibling* spans under the same parent, never as re-emissions of the original `span_id`. 3. Downstream aggregation deduplicates on `(trace_id, span_id)` before summing cost or tokens, and treats `(trace_id, semantic_role)` as a non-unique grouping key. 4. The orchestrator records an explicit `attempt` attribute on each retried span so that aggregation can select the terminal attempt when computing billed cost. When any of these fails, `trace_id` degrades from a correlation key to a mere bucket label, and any metric derived by grouping on it inherits the duplication. ## Decision Boundary The boundary is crossed at the moment an agent retries a child call *within* the same trace rather than opening a new trace or a new span. Concretely: - **Below the boundary:** retries are modeled as sibling spans with distinct `span_id` and an `attempt` attribute; `trace_id` remains a valid correlation key and joins are safe. - **At the boundary:** retries reuse the parent `trace_id` but the writer does not emit `attempt` metadata. Correlation still works for *presence* queries ("did the retriever run?") but fails for *aggregate* queries ("how many tokens did the retriever consume?"). - **Above the boundary:** the writer reuses both `trace_id` and `span_id` across attempts. Now the trace store contains overwrites or duplicate primary keys, and the correlation key is no longer even a bucket label — it is corrupted. This is the condition under which an instance *must not* treat `trace_id` as an identity, and must escalate to the trace store's dedup layer. The decision rule: if a span consumer cannot prove conditions 1–4 hold for a given trace, it must treat `trace_id` as non-unique and fall back to `(trace_id, span_id, attempt)` joins, or refuse to emit cost/token aggregates for that trace. ## Operational Effect Instances that skip this check produce observability dashboards whose cost and latency numbers are systematically biased upward on any trace containing a retry — and retries are concentrated precisely on the slow, failing paths that operators care most about. The bias is not uniform: it scales with retry count, so the worst-affected traces are the ones most likely to be inspected during incident response. The practical mitigation is to make `attempt` a first-class span attribute at emission time and to gate aggregate computation on its presence, rather than attempting to reconstruct attempt boundaries post hoc from timestamps (which fails under clock skew across agents). ```python from dataclasses import dataclass class TraceCorrelationError(RuntimeError): """Raised when a trace_id is used as an identity but retry structure is unproven.""" pass @dataclass(frozen=True) class Span: trace_id: str span_id: str semantic_role: str attempt: int | None prompt_tokens: int duration_ms: float def aggregate_tokens(spans: list[Span], trace_id: str) -> int: """Sum billed prompt tokens for a trace, deduplicating retried attempts. Treats trace_id as a correlation key, not an identity. Requires that every span in the trace carries an explicit attempt index; otherwise the trace's retry structure is unproven and aggregation would double-count. """ trace_spans = [s for s in spans if s.trace_id == trace_id] if not trace_spans: return 0 if any(s.attempt is None for s in trace_spans): raise TraceCorrelationError( f"trace {trace_id!r} contains spans without attempt metadata; " "trace_id cannot be treated as an identity for token aggregation" ) # Keep only the terminal attempt per (semantic_role, parent) grouping key. terminal: dict[str, Span] = {} for s in trace_spans: key = s.semantic_role prev = terminal.get(key) if prev is None or s.attempt > prev.attempt: terminal[key] = s return sum(s.prompt_tokens for s in terminal.values()) def assert_unique_span_ids(spans: list[Span]) -> None: seen: set[tuple[str, str]] = set() for s in spans: key = (s.trace_id, s.span_id) if key in seen: raise TraceCorrelationError( f"duplicate (trace_id, span_id) = {key!r}; " "span_id was reused across attempts, trace store is corrupted" ) seen.add(key) ```