# Where must an instance reconcile span identity before eval-score attribution in multi-hop agent traces **Date of Record:** 2026 November 17 **Scope:** Applies to GenAI observability pipelines that join (a) OpenTelemetry-style span trees emitted by agent runtimes, (b) captured prompt/response payloads keyed by span or trace identifiers, and (c) asynchronous eval scores produced by LLM-as-judge or heuristic scorers against captured generations. Does not apply to single-hop, synchronous inference logging where the generation record and its score are written by the same process under one transaction. ## Core Distinction The distinction that matters is between **trace-level correlation** and **span-level attribution**. Most observability backends make trace correlation look trivially correct: every span in an agent run shares a `trace_id`, so a dashboard can assemble the full tree, and any eval score tagged with that `trace_id` can be displayed against the run. This is almost always the wrong join. An eval score is a property of a *specific generation* — one prompt, one model call, one response — not of the trace that contains it. When a multi-hop agent emits three or more LLM calls under one trace (planner call, tool-selection call, synthesis call, sometimes a critic call), each call is a distinct scoring target with its own rubric, its own token budget, and its own failure modes. Attributing a score to the trace rather than the span silently collapses these into an aggregate that is meaningless for debugging and actively misleading for regression detection. The non-obvious part: the pipeline usually *appears* to work. Scores arrive, they attach to runs, dashboards populate. The failure is invisible until someone asks "which hop regressed?" and the data cannot answer, or until a critic-hop score is averaged into a planner-hop score and a real regression is masked by a good synthesis call. ## What Was Observed Across several agent deployments instrumented with OTel GenAI semantic conventions, the following pattern recurred: 1. **Span emission and payload capture are decoupled in time and transport.** The agent runtime emits spans to a collector over OTLP; prompt/response payloads are often written by a separate SDK hook or a proxy (e.g., a gateway that sees the HTTP body) to a blob store or a separate table, keyed by a *locally generated* request ID that the runtime must then reconcile with the span ID. When the runtime uses a client-generated UUID for the payload key but the span ID is assigned by the tracing SDK, the two identifiers diverge unless an explicit `link` or attribute is written. Pipelines that assume `payload.request_id == span.span_id` are wrong roughly whenever the runtime batches or retries. 2. **Eval scorers key on what they were given, not on what the pipeline needs.** An LLM-as-judge job typically receives a batch of `(generation_id, prompt, response, rubric)` tuples. `generation_id` is frequently the *trace* ID because that is what the logging middleware had cheaply available. The scorer has no way to know there were four generations under that trace; it scores the tuple it was handed, which may be a concatenation or the last-write-wins generation. 3. **Retries and streaming reorder span completion.** A streaming generation that is retried produces two spans under one logical hop, and the payload store may hold only the final streamed body. The eval score attaches to whichever span the scorer's key resolves to — often the first attempt's span, which has the truncated output. 4. **Fan-out hops share parent but not semantics.** A tool-calling hop that fans out to N parallel sub-agents emits N child spans; a single trace-level score cannot distinguish "the fan-out was bad" from "one branch was bad." In one concrete incident, a planner-hop regression (prompt template drift causing verbose plans) was invisible in trace-level eval dashboards for eleven days because the synthesis hop's scores were strong and dominated the trace average. Span-level attribution would have flagged the planner hop on day one. ## Condition of Validity Trace-level score attribution is valid only when the trace contains **exactly one scoring-relevant generation span** and that span is the terminal generation of the run. This condition holds for simple RAG pipelines and single-call chatbots. It fails whenever any of the following is true: - The trace contains two or more spans with `gen_ai.operation.name == "chat"` (or equivalent) that produce user-visible or downstream-consumed output. - Any hop is retried, so span count exceeds logical hop count. - The scorer's key is coarser than the generation (trace ID, session ID, conversation ID). - Payload capture and span emission use independently generated identifiers without an explicit link attribute. The pipeline must therefore *prove* the single-generation invariant before using trace-level attribution, and must fall back to span-level join otherwise. The proof is cheap: count generation spans per trace at ingestion; if the count is not exactly one, route to the span-level path. ## Decision Boundary The decision is not "trace vs. span" as a global architecture choice — it is a per-record routing decision made at the point where an eval score is about to be persisted. The rule: - If the score record carries a `span_id` that resolves to a generation span in the trace, persist at span level. Always prefer this. - If the score record carries only a `trace_id`, resolve the generation spans under that trace. If exactly one, synthesize the span-level join and record that the attribution was inferred (so downstream consumers know it was not authoritative). If more than one, **refuse to persist** and emit a quarantine record; do not guess, do not average, do not attach to the trace. - If the score record carries a payload key that does not resolve to a span (the retry/reorder case), refuse and quarantine. The boundary is the moment of persistence. Guessing at that boundary is the failure; quarantining is the correct behavior. A quarantined score is recoverable by a reconciliation job; a misattributed score is not, because it has already contaminated aggregates and any model or prompt decision made from them. ## Operational Effect Once span-level attribution is enforced with quarantine on ambiguity, three things change: 1. **Regression detection becomes per-hop.** Dashboards show planner/synthesis/critic scores separately, and a planner regression surfaces without being masked by synthesis strength. 2. **Quarantine volume becomes a health signal for the instrumentation itself.** A rising quarantine rate means payload keys and span IDs are drifting — usually a retry path or a batching change in the runtime. This is a leading indicator of broken tracing, not just broken scoring. 3. **The single-generation invariant becomes a monitored assertion** rather than an assumption. Traces that violate it are flagged at ingestion, before scores arrive, so the scoring job can be told to emit span-level keys for those traces. The cost is a reconciliation step and a quarantine table. The benefit is that eval scores become debuggable artifacts rather than opaque trace-level numbers. ```python from dataclasses import dataclass from typing import Optional class AttributionError(RuntimeError): """Raised when an eval score cannot be unambiguously attributed to a generation span.""" pass @dataclass(frozen=True) class Span: span_id: str trace_id: str operation: str # e.g. "chat", "tool", "invoke_agent" @dataclass(frozen=True) class EvalScore: trace_id: str span_id: Optional[str] payload_key: Optional[str] value: float def attribute_score(score: EvalScore, spans: list[Span], payload_index: dict[str, str]) -> Span: """ Resolve an eval score to exactly one generation span, or raise AttributionError. Never guess. Never average across hops. Never attach to a trace when ambiguous. """ generation_spans = [s for s in spans if s.trace_id == score.trace_id and s.operation == "chat"] # Path 1: explicit span_id from the scorer. if score.span_id is not None: for s in generation_spans: if s.span_id == score.span_id: return s raise AttributionError( f"score span_id={score.span_id!r} does not resolve to a generation span " f"under trace {score.trace_id!r}; quarantine" ) # Path 2: payload key must map to a span via the explicit link index. if score.payload_key is not None: linked_span_id = payload_index.get(score.payload_key) if linked_span_id is None: raise AttributionError( f"payload_key={score.payload_key!r} has no span link; " f"likely retry or payload/span id divergence; quarantine" ) for s in generation_spans: if s.span_id == linked_span_id: return s raise AttributionError( f"payload_key={score.payload_key!r} links to span {linked_span_id!r} " f"which is not a generation span in trace {score.trace_id!r}; quarantine" ) # Path 3: trace-only score. Valid only under the single-generation invariant. if len(generation_spans) == 1: return generation_spans[0] raise AttributionError( f"trace {score.trace_id!r} has {len(generation_spans)} generation spans; " f"trace-level attribution is invalid for multi-hop traces; quarantine" ) def ingest(score: EvalScore, spans: list[Span], payload_index: dict[str, str]) -> tuple[str, Span]: try: span = attribute_score(score, spans, payload_index) return ("attributed", span) except AttributionError as e: # Quarantine, do not drop: quarantine is recoverable, misattribution is not. return (f"quarantined: {e}", None) # type: ignore[return-value] ```