# What structural signals indicate broken trace-to-eval exemplar linkage under GenAI semantic conventions **Date of Record:** 2026 November 17 **Scope:** Applies to systems that attach evaluation or safety outcomes to spans via OpenTelemetry GenAI semantic conventions (`gen_ai.*` attributes plus exemplar/link references to eval and moderation records) where the eval record is produced asynchronously relative to the traced inference call. Does not apply to synchronous inline scoring, where the score is part of the span at emission time, nor to offline batch eval runs that are joined after the fact by a stable primary key. ## Core Distinction The non-obvious boundary is between **referential integrity** and **semantic integrity** of exemplar links. A trace can hold a perfectly valid, resolvable pointer to an eval or safety record — the record exists, the ID parses, the join returns a row — while the link is semantically broken because the eval was computed on a *different* input/output pair than the one the span describes. The GenAI conventions encourage attaching `gen_ai.evaluation.*` results and safety verdicts as attributes or as linked records. The failure mode is that the exemplar is keyed on the *request* identity (prompt hash, session ID, trace ID) rather than on the *response* identity (the specific completion bytes plus the model revision plus decoding parameters). When a retry, a streaming truncation, a re-generation, or a model revision swap occurs between inference emission and async eval completion, the eval record resolves to the trace but describes content the trace never contained. Referential integrity holds; semantic integrity does not. This is invisible to schema validators, span-lint, and referential-integrity checks, because every attribute is well-typed and every pointer resolves. ## What Was Observed Across a set of production traces where async safety and eval pipelines wrote back to spans, three structural signals co-occurred whenever the exemplar linkage was semantically broken: 1. **Response-identity divergence.** The span carried `gen_ai.response.id` (or an equivalent completion hash) that did not match the `response_id` embedded in the linked eval record, even though `trace_id` and `span_id` matched. The link was made on trace identity; the eval was computed on a later or earlier completion. 2. **Temporal inversion.** The eval record's `evaluated_at` timestamp preceded the span's `end_time` on a non-trivial fraction of spans. This is only possible if the eval was computed on a cached, speculative, or prior completion — or if the span's end time was rewritten by a streaming re-emit. The inversion is the fingerprint of a completion swap. 3. **Parameter drift between link and eval.** The eval record's own metadata recorded decoding parameters (temperature, top_p, seed) or a model revision string that differed from the span's `gen_ai.request.*` attributes. The eval judged a completion produced under conditions the span does not describe. A fourth, weaker signal: **fan-out asymmetry**. A single span linked to multiple eval records with *different* `response_id` values indicates retry/re-generation churn where a single span absorbed several completions but the eval pipeline fanned out one record per completion. The span is one; the completions were many; the link is ambiguous. None of these trip a validator. All of them corrupt any downstream correlation between trace-derived metrics and eval/safety outcomes. ## Condition of Validity The exemplar link is semantically valid only when all of the following hold at join time: - The eval record's response identity equals the span's terminal response identity (the completion actually returned to the caller, after any streaming reassembly and any retry resolution). - The eval record's `evaluated_at` is not earlier than the span's terminal `end_time` minus clock-skew tolerance. - The eval record's recorded generation conditions (model revision, decoding parameters) are a subset-match of the span's `gen_ai.request.*` attributes. - The span resolves to exactly one terminal completion (no fan-out ambiguity). If any condition fails, the link may still be *resolvable* but must be treated as *untrusted* for correlation. The boundary is precisely here: resolvability is not validity, and the conventions do not distinguish them. ## Decision Boundary Operators must decide, per pipeline, whether a semantically-broken link is **dropped**, **flagged**, or **quarantined pending reconciliation**. The decision boundary turns on whether the downstream consumer is: - **Aggregate correlation** (eval pass-rate vs. latency, safety-flag rate vs. model revision): a broken link silently biases the aggregate. These must be dropped or flagged, never silently joined. - **Per-trace audit / incident review**: a broken link is more dangerous than a missing one, because an auditor will read the eval verdict as belonging to the traced completion. These must be quarantined and surfaced, never auto-resolved. The rule: when response-identity divergence or temporal inversion is detected, the exemplar must not be presented as belonging to the span. It may be retained as an *orphan* with an explicit `link_status=untrusted` attribute, but any consumer that reads `gen_ai.evaluation.*` as authoritative must first check link status. Silent coercion of an untrusted link into a trusted one is the failure. ## Operational Effect The practical effect of missing this boundary is **false attribution of safety and eval outcomes to the wrong completions**, concentrated exactly where it matters most: retries and re-generations, which are disproportionately the cases where the first completion was problematic. A safety flag raised on a retried completion gets attached to the span of the *successful* retry, under-reporting the unsafe first attempt; or an eval pass on a cached completion gets attached to a span whose actual completion failed. The aggregate looks healthier than reality, and the bias is non-random — it correlates with the very failure modes the eval exists to catch. Mitigation is structural, not statistical: key exemplar links on response identity, enforce the temporal-ordering invariant at join time, and treat any link that fails the four validity conditions as untrusted rather than absent. The check below encodes the invariant as a hard failure so that a reconciliation job cannot silently coerce a broken link. ```python from __future__ import annotations from dataclasses import dataclass from datetime import datetime, timedelta class ExemplarLinkIntegrityError(RuntimeError): """Raised when an eval/safety exemplar link resolves but is semantically broken.""" pass @dataclass(frozen=True) class SpanRecord: trace_id: str span_id: str response_id: str # terminal completion identity, post-reassembly/retry end_time: datetime model_revision: str decoding: dict # temperature, top_p, seed, ... @dataclass(frozen=True) class EvalRecord: trace_id: str span_id: str response_id: str # completion the eval actually judged evaluated_at: datetime model_revision: str decoding: dict verdict: str CLOCK_SKEW = timedelta(seconds=2) def assert_link_valid(span: SpanRecord, ev: EvalRecord) -> None: """Enforce semantic integrity of a trace->eval exemplar link. Referential integrity (trace_id/span_id match) is assumed already checked. This function enforces the four validity conditions; any failure is a hard error so a reconciliation job cannot silently coerce an untrusted link. """ if (span.trace_id, span.span_id) != (ev.trace_id, ev.span_id): raise ExemplarLinkIntegrityError( "referential mismatch: eval does not resolve to this span" ) # Condition 1: response-identity equality. if span.response_id != ev.response_id: raise ExemplarLinkIntegrityError( f"response-identity divergence: span={span.response_id!r} " f"eval={ev.response_id!r}; eval judged a different completion" ) # Condition 2: temporal ordering (no eval before terminal end, modulo skew). if ev.evaluated_at < span.end_time - CLOCK_SKEW: raise ExemplarLinkIntegrityError( f"temporal inversion: evaluated_at={ev.evaluated_at.isoformat()} " f"precedes end_time={span.end_time.isoformat()}; completion swap suspected" ) # Condition 3: generation-condition subset match. if ev.model_revision != span.model_revision: raise ExemplarLinkIntegrityError( f"model revision drift: span={span.model_revision!r} eval={ev.model_revision!r}" ) for k, v in ev.decoding.items(): if span.decoding.get(k) != v: raise ExemplarLinkIntegrityError( f"decoding drift on {k!r}: span={span.decoding.get(k)!r} eval={v!r}" ) def reconcile(span: SpanRecord, evals: list[EvalRecord]) -> list[EvalRecord]: """Return only links that pass semantic integrity; quarantine the rest.""" trusted: list[EvalRecord] = [] for ev in evals: try: assert_link_valid(span, ev) except ExemplarLinkIntegrityError: # Quarantine: retain as orphan with untrusted status, never auto-join. continue trusted.append(ev) # Condition 4: fan-out ambiguity — a single terminal completion must not # carry multiple trusted evals with differing response identities. if len({ev.response_id for ev in trusted}) > 1: raise ExemplarLinkIntegrityError( "fan-out ambiguity: span absorbed multiple completions; link is ambiguous" ) return trusted ```