# What structural signals indicate when GenAI observability exemplars stop linking traces to eval regressions **Date of Record:** 2026 November 15 **Scope:** Applies to systems that attach trace exemplars (span IDs, request hashes, prompt fingerprints) to eval regression reports so that an operator can jump from a failing eval metric to the production trace that triggered it. Does not apply to offline eval suites with no production trace linkage, nor to pure log-based observability without exemplar attachment. ## Core Distinction The non-obvious boundary is between *exemplar presence* and *exemplar referential integrity*. Most observability stacks treat "the eval regression has N exemplars attached" as success. But an exemplar is a pointer, not a payload. The invariant that actually matters is: > Every exemplar attached to an eval regression must resolve, at read time, to a trace record whose request fingerprint and model/config version are byte-identical to those captured at eval-generation time. This invariant fails silently. The regression report still renders, the exemplar count is still non-zero, and dashboards stay green. What breaks is the *join*: the trace store has rotated, the model version has been promoted, the prompt template hash has drifted, or the span was sampled out of retention. The exemplar becomes a dangling foreign key into an append-only store that no longer contains the row. ## What Was Observed Across three incidents, the following structural signals preceded the discovery that exemplar linkage had been broken for 11–34 days: 1. **Fingerprint drift without count drift.** Exemplar *counts* per regression remained stable, but the distribution of `prompt_template_hash` values attached to exemplars no longer matched the distribution of hashes in the live trace store. The eval harness was re-emitting exemplars from a cached batch computed against template v41, while production had moved to v43. 2. **Trace-ID resolution latency cliff.** Resolving an exemplar's trace ID went from p99 < 40 ms to timeouts, not because the trace store was slow, but because the ID namespace had been re-sharded and old IDs now required a cross-shard fallback lookup that was never provisioned. 3. **Orphan ratio crossing a threshold.** The ratio of exemplars whose `(trace_id, span_id)` pair returned 404 on resolution crossed ~0.5% — small in absolute terms, but structurally significant because it indicated retention windows had begun evicting traces *younger* than the eval regression window. 4. **Config-version skew in the exemplar envelope.** Exemplars carried `model_config_version` at generation time but the resolution layer resolved against the *current* config, producing a false match: the trace resolved, but it was a different request that happened to reuse the span ID after ID recycling. The last signal is the dangerous one. A 404 is loud. A *false positive resolution* — an exemplar that resolves to the wrong trace — is silent and produces spurious "root cause" attributions during incident review. ## Condition of Validity Exemplar linkage is valid only while all of the following hold simultaneously: - `trace_retention_window > eval_regression_window + max_eval_lag`. If evals are computed with a 24h lag over a 7d window, retention must exceed 8d with margin. - Span IDs are never recycled within the retention window (monotonic or sufficiently wide ID space). - The exemplar envelope stores the full resolution key: `(trace_id, span_id, prompt_template_hash, model_config_version, sampling_seed)`. Storing only `trace_id` is insufficient because it does not let the resolver detect config skew. - The resolver validates the resolved trace's fingerprint against the exemplar's fingerprint and raises on mismatch rather than returning the trace. If any of these is violated, exemplar *presence* is not evidence of exemplar *validity*. ## Decision Boundary When the orphan ratio exceeds a configured threshold OR fingerprint-match failures appear during resolution, the correct action is **not** to widen retention blindly (cost) and **not** to suppress the exemplar (loses the eval signal). The boundary decision is: - If orphans are concentrated in a *time band* (recent traces evicted), the fault is retention/window configuration → fix retention, do not touch the eval pipeline. - If orphans are concentrated in a *fingerprint band* (a specific template or config version), the fault is a deploy that changed the fingerprint without re-emitting exemplars → invalidate exemplars by fingerprint, do not widen retention. - If resolution returns a trace with a *mismatched fingerprint*, treat as data corruption: halt eval-to-trace attribution until the ID-recycling or resolver bug is fixed. This is the only case where the pipeline must stop. Conflating these three produces the classic mis-fix: widening retention to cure a fingerprint-drift problem, which only delays detection. ## Operational Effect Teams that rely on exemplar counts rather than resolution success will ship eval regressions to on-call without a resolvable trace, converting a 5-minute root-cause lookup into a 2-hour archaeology session over rotated logs. The structural fix is to make resolution a first-class, validated operation that fails loudly on fingerprint mismatch, and to monitor the orphan ratio and fingerprint-distribution divergence as leading indicators — not the exemplar count. ```python import hashlib from dataclasses import dataclass class ExemplarLinkageError(RuntimeError): """Raised when an eval-regression exemplar cannot be validly resolved to a trace.""" pass @dataclass(frozen=True) class Exemplar: trace_id: str span_id: str prompt_template_hash: str model_config_version: str @dataclass(frozen=True) class TraceRecord: trace_id: str span_id: str prompt_template_hash: str model_config_version: str def resolve_exemplar(exemplar: Exemplar, trace_store: dict) -> TraceRecord: key = (exemplar.trace_id, exemplar.span_id) record = trace_store.get(key) if record is None: raise ExemplarLinkageError( f"orphan exemplar: no trace for key={key}; " f"likely retention eviction or re-sharding" ) if record.prompt_template_hash != exemplar.prompt_template_hash: raise ExemplarLinkageError( f"fingerprint drift on {key}: exemplar template " f"{exemplar.prompt_template_hash} != resolved " f"{record.prompt_template_hash}; treat as config-skew, " f"not retention fault" ) if record.model_config_version != exemplar.model_config_version: raise ExemplarLinkageError( f"config-version skew on {key}: exemplar {exemplar.model_config_version} " f"!= resolved {record.model_config_version}; possible span-ID recycling " f"— halt attribution until resolver is fixed" ) return record def orphan_ratio(exemplars, trace_store) -> float: if not exemplars: return 0.0 orphans = 0 for ex in exemplars: if (ex.trace_id, ex.span_id) not in trace_store: orphans += 1 return orphans / len(exemplars) ```