# When must an instance resolve trace-eval-safety joins at agent-step granularity rather than at session or request granularity **Date of Record:** 2026 November 17 **Scope:** Applies to GenAI observability pipelines that ingest OpenTelemetry-style agent traces (span trees with `agent.step` spans), asynchronous evaluation verdicts (LLM-as-judge, rubric scorers, heuristic checks), and safety classifier signals (toxicity, prompt-injection, PII, jailbreak) that arrive out of band with independent latency and revision semantics. Does not apply to single-turn, stateless inference endpoints where one request maps to exactly one trace and one response, nor to batch offline scoring where no live correlation is required. ## Core Distinction The non-obvious boundary is between **join key identity** and **join key stability**. At session granularity, the join key is the session ID — immutable, assigned once, stable across the entire lifetime of the interaction. At agent-step granularity, the join key is `(trace_id, span_id, step_index)`, and the critical property is that `span_id` is stable but `step_index` is *not*: retries, re-planning, tool-call fan-out, and parallel sub-agent branches all produce multiple spans that a naive pipeline will renumber, reorder, or collapse. The failure mode is not "the join fails" — it is "the join succeeds against the wrong step." A safety verdict for step 3 lands on the span currently labeled step 3 after a retry shifted the numbering, and the pipeline emits a clean, well-formed, silently wrong correlation. The distinction matters because evaluation and safety signals are *late-bound and revisable* in a way that traces are not. A trace span is append-only. An eval verdict is not: judges get re-run with a new model version, safety classifiers get re-calibrated, human reviewers override automated flags. If you join at session granularity, a revised verdict simply overwrites the session-level score — coarse but correct. If you join at step granularity without a revision-aware key, a revised verdict for the *original* step 3 can land on the *retried* step 3 and corrupt two steps' worth of signal at once. ## What Was Observed In a production pipeline correlating ~40M agent steps/day, we observed a class of incident that never triggered an alert: **step-attributed safety flags whose span ancestry did not match the span that generated the flagged content.** The pipeline's join used `(trace_id, step_index)` because `step_index` was cheap to extract from the eval payload and `span_id` required a second lookup against the trace store. Under normal operation these agreed. Under retries they did not. Concretely: an agent step that called an external tool, received a malformed response, and retried produced two spans with the same logical `step_index` but different `span_id`s — the failed attempt and the successful retry. The tool output in the failed attempt contained injected instructions; the retry did not. The safety classifier, running against the *failed attempt's* payload (correctly — it should flag injected content regardless of whether the agent acted on it), emitted a verdict keyed to `(trace_id, step_index=7)`. The join resolved it to the retry span. The dashboard showed a clean step. The injection signal was lost. The inverse also occurred: a benign retry inherited a safety flag from a failed attempt whose payload it had never seen, inflating false-positive rates on a customer-facing trust metric by roughly 3x for affected traces. Neither direction raised an error. Both produced structurally valid records. ## Condition of Validity Step-granularity joining is only valid when **all three** of the following hold: 1. **The join key includes `span_id`, not merely `step_index`.** `step_index` is a presentation attribute, not an identity. Any pipeline that reconstructs step identity from ordinal position is unsound under retry, fan-out, or branch. 2. **Late-bound signals carry an explicit revision and a target-span reference that is resolved against the trace store, not against a locally maintained ordinal map.** The eval payload must name the `span_id` it scored; the pipeline must verify that span exists and that its payload hash matches what was scored before attaching the verdict. 3. **Fan-out and parallel branches are represented as distinct spans with distinct `span_id`s, and the join is one-to-many, not one-to-one.** A single logical step that fans out to three tool calls produces three spans; a step-level safety verdict must attach to all three (or to their common parent) explicitly, not to whichever span the join happens to hit first. When any of these fails, the correct granularity is coarser — session or request — where the join key is stable and a misattribution degrades gracefully to a coarser but honest signal. ## Decision Boundary Resolve at **agent-step granularity** when: the trace store is authoritative and queryable at join time; eval and safety payloads carry `span_id` and a payload hash; revisions are modeled as new verdict records rather than in-place updates; and the pipeline can detect ordinal drift (retry, renumber) at ingest. Fall back to **session granularity** when: any signal source cannot supply `span_id`; the trace store is lossy or sampled such that target spans may be absent; or the agent framework renumbers steps without emitting stable span identity. In these cases, step-level attribution is not "less precise" — it is actively wrong, and a session-level aggregate is the only defensible record. Do **not** attempt to reconstruct `span_id` from `step_index` plus heuristics (timestamps, payload similarity). We tried; payload similarity matched the failed attempt to the retry in 11% of retried steps, which is worse than no join because it produces confident misattribution. ## Operational Effect After switching to `span_id`-keyed joins with payload-hash verification and revision-as-append, step-attributed safety recall on retried steps recovered to parity with non-retried steps, and the trust-metric false-positive inflation collapsed. The cost was a mandatory trace-store lookup per late-bound verdict, which we absorbed by caching span payload hashes at ingest. The lesson generalizes: **in observability pipelines, the join key's stability under retry determines whether a fine-grained correlation is signal or noise, and this is invisible in the happy path because the happy path has no retries.** ```python from __future__ import annotations from dataclasses import dataclass, field from typing import Optional class StepJoinAttributionError(RuntimeError): """Raised when a late-bound eval/safety verdict cannot be soundly attributed to a specific agent step.""" @dataclass(frozen=True) class SpanRecord: trace_id: str span_id: str step_index: int payload_hash: str is_retry: bool = False @dataclass(frozen=True) class Verdict: trace_id: str target_span_id: Optional[str] scored_payload_hash: Optional[str] step_index_hint: int kind: str # "eval" | "safety" revision: int = 1 @dataclass class StepJoinResolver: spans_by_id: dict[str, SpanRecord] = field(default_factory=dict) spans_by_ordinal: dict[tuple[str, int], list[SpanRecord]] = field(default_factory=dict) def register(self, span: SpanRecord) -> None: self.spans_by_id[span.span_id] = span self.spans_by_ordinal.setdefault((span.trace_id, span.step_index), []).append(span) def resolve(self, verdict: Verdict, min_granularity: str = "step") -> SpanRecord: if min_granularity != "step": raise StepJoinAttributionError( "caller requested session-granularity fallback; do not call step resolver" ) # Rule 1: identity must be span_id, never ordinal. if verdict.target_span_id is None: raise StepJoinAttributionError( f"verdict for trace {verdict.trace_id} step {verdict.step_index_hint} " f"has no target_span_id; ordinal-only join is unsound under retry" ) span = self.spans_by_id.get(verdict.target_span_id) if span is None: raise StepJoinAttributionError( f"target span {verdict.target_span_id} absent from trace store; " f"cannot verify attribution (sampling or loss)" ) # Rule 2: payload hash must match what was actually scored. if verdict.scored_payload_hash is None: raise StepJoinAttributionError( f"verdict {verdict.kind} rev {verdict.revision} lacks payload hash; " f"revision-aware attribution impossible" ) if verdict.scored_payload_hash != span.payload_hash: raise StepJoinAttributionError( f"payload hash mismatch for span {span.span_id}: " f"scored {verdict.scored_payload_hash} != stored {span.payload_hash}; " f"likely stale verdict against a prior attempt" ) # Rule 3: detect ordinal drift and refuse silent collapse. siblings = self.spans_by_ordinal.get((span.trace_id, span.step_index), []) if len(siblings) > 1 and span.is_retry is False: raise StepJoinAttributionError( f"ordinal drift detected: step {span.step_index} in trace {span.trace_id} " f"has {len(siblings)} spans; step-level verdict must name span_id explicitly" ) return span ```