# Which invariants prevent per-turn trace/eval/safety joins from silently misattributing verdicts across retries, speculative drafts, and streaming partials **Date of Record:** 2026 November 17 **Scope:** Applies to GenAI observability pipelines that emit one logical "agent turn" record by joining a trace span, one or more evaluation verdicts, and one or more safety signals keyed by turn identity. Does not apply to batch offline eval systems where turn identity is assigned after the fact by a single writer, nor to pipelines where evals and safety classifiers are computed inline in the same process that owns the span. ## Core Distinction The failure mode is not "missing data" or "orphaned verdicts." It is **verdict misattribution under turn-identity reuse**. A turn is a logical unit — one user input, one agent response, possibly many model calls. The trace span, the eval run, and the safety classifier each observe the turn through a *different* physical substrate: the span observes an execution attempt, the eval observes a rendered transcript, the safety signal observes a token stream (or a post-hoc text). Each substrate has its own notion of "which attempt am I," and those notions desynchronize under three routine conditions: retries (a second attempt at the same logical turn), speculative drafts (candidate continuations that are discarded), and streaming partials (verdicts computed on prefixes that never become the final text). The invariant that is usually assumed — *turn_id uniquely identifies the content being judged* — is false. The invariant that actually holds is weaker: *turn_id identifies the logical turn; attempt_id identifies the content*. Pipelines that join on turn_id alone are correct only in the absence of retries, speculation, and prefix-level scoring. All three are present in production agent stacks. ## What Was Observed Across three deployed pipelines, the same class of defect appeared with different surface symptoms: 1. **Retry shadowing.** An agent turn issued a model call, the call timed out at the gateway, and the orchestrator retried. The trace recorded two spans, both carrying the same `turn_id` and a `retry_index` attribute on the second. The eval service, however, was triggered by a post-turn transcript event and scored only the *final* transcript, emitting verdict `V_final` under `turn_id`. The safety classifier was triggered per model call and emitted `S_attempt1` (flagged) and `S_attempt2` (clean), both under `turn_id`. The join produced a turn record with a safety flag from the failed attempt and an eval verdict from the successful attempt — a record that describes no real execution. Downstream, the flagged-but-clean turn was counted as a safety incident with a passing eval, inflating the "safe but wrong" quadrant of the eval dashboard. 2. **Speculative draft contamination.** A decoding setup produced two candidate continuations (`attempt_id` 0 and 1) for the same turn; the safety classifier scored both because it ran on the token stream before selection. Only candidate 1 was committed. The classifier emitted `S_0` (blocked) and `S_1` (allowed). The span and eval carried only the committed attempt. A join keyed on `turn_id` collapsed both safety signals into one record; depending on join order (first-wins vs. last-wins vs. aggregate), the turn was reported as blocked, allowed, or "conflicting." The aggregate policy was never specified, so three services disagreed about the same turn. 3. **Streaming partial verdicts.** The safety classifier ran on 256-token windows of a streaming response and emitted a verdict per window. The final text was 1,900 tokens; the classifier emitted seven verdicts, one of which (window 3, a false positive on a code snippet) was flagged. The turn-level join took the max-severity verdict, producing a flagged turn whose committed text contains nothing the classifier would flag on a full read. The eval, run on the full text, passed. The resulting record — "safety flagged, eval passed" — was indistinguishable from a genuine near-miss. In all three cases the pipeline did not crash, did not drop data, and did not report a join error. It produced a well-formed record that was **internally inconsistent with any single execution attempt**. ## Condition of Validity The join is sound if and only if all of the following hold simultaneously: - **Content-addressed verdicts.** Every eval verdict and safety signal carries the hash (or immutable content id) of the exact text it judged, not merely the `turn_id`. A verdict whose content hash does not match the committed attempt's content hash is *not* a verdict about the committed attempt. - **Attempt-scoped emission.** Evals and safety classifiers emit under `(turn_id, attempt_id, content_hash)`, and the join is performed on the committed `(turn_id, attempt_id)` only. Non-committed attempts are retained as *diagnostic* records, never merged into the committed turn. - **Window provenance.** Streaming verdicts carry `(window_start, window_end, text_hash_of_window)` and a declared rollup policy (max-severity, any-flag, last-window, full-text-recheck). The rollup policy is part of the turn record, not implicit in the join. - **Deterministic commit marker.** The orchestrator emits an explicit `committed_attempt_id` (or a `commit_event`) that the join consumes. Absent a commit marker, the join must not guess from arrival order or retry index. When any of these is absent, the join is *undefined*: it will produce a record, but the record's semantics are join-order-dependent. The dangerous property is that undefined joins do not fail loudly — they fail *plausibly*. ## Decision Boundary The operational rule is a hard split on whether `committed_attempt_id` and content hashes are present at join time: - **If `committed_attempt_id` and per-verdict content hashes are present** → perform the strict join; any verdict whose content hash mismatches the committed attempt is quarantined to a `misattributed_verdicts` side channel with a reason code, and the turn record is marked `join_complete=true`. - **If `committed_attempt_id` is absent** → do **not** emit a turn record. Emit a `pending_commit` record with the raw attempts and verdicts attached, and let a reconciler resolve it once the commit event arrives (or mark it `unresolvable` after a bounded TTL). Never select an attempt by retry index, arrival time, or "last writer." - **If content hashes are absent but `attempt_id` is present** → join is permitted only when exactly one attempt exists for the turn; otherwise treat as the pending-commit case. - **If streaming window provenance is absent** → the safety signal may be attached as `signal_scope=unknown` and must not contribute to the turn-level severity rollup; it is visible but non-authoritative. The boundary is therefore not "is the data complete" but "is the content identity of each verdict pinned to the content identity of the committed attempt." Pinning is the precondition; completeness is not. ## Operational Effect Adopting the strict join changes three things in practice. First, the "safe but wrong" and "flagged but passing" quadrants of the eval dashboard shrink to genuine cases, because misattributed verdicts are quarantined rather than merged. Second, the pipeline gains a measurable `misattribution_rate` (quarantined verdicts / total verdicts) that is a leading indicator of retry storms, speculation churn, and classifier windowing problems — a signal that was previously invisible. Third, turn records acquire a `join_complete` field that downstream consumers must respect; dashboards that ignore it will undercount turns during retry storms, which is the correct behavior (better to show `pending_commit` than a fabricated record). The cost is latency: turns with retries or speculation wait for the commit marker rather than emitting eagerly. In practice the commit marker arrives within the same orchestration step, so the added latency is bounded by the reconciler TTL, not by model latency. The invariant to encode is: **a turn record is a statement about one committed attempt, and no verdict enters it without proving which attempt it judged.** ```python from __future__ import annotations from dataclasses import dataclass, field from typing import Optional class MisattributedVerdictError(RuntimeError): """Raised when a verdict's content identity cannot be pinned to the committed attempt of a turn, and the join is not permitted to guess.""" @dataclass(frozen=True) class Verdict: turn_id: str attempt_id: Optional[int] content_hash: Optional[str] kind: str # "eval" | "safety" severity: int # 0 = clean, higher = worse scope: str = "attempt" # "attempt" | "window" | "unknown" @dataclass class TurnCommit: turn_id: str committed_attempt_id: int committed_content_hash: str @dataclass class TurnRecord: turn_id: str committed_attempt_id: int verdicts: list[Verdict] = field(default_factory=list) quarantined: list[Verdict] = field(default_factory=list) join_complete: bool = True def join_turn(commit: Optional[TurnCommit], verdicts: list[Verdict]) -> TurnRecord: if commit is None: # No commit marker: refuse to select an attempt by arrival order. raise MisattributedVerdictError( "no committed_attempt_id for turn; emit pending_commit instead of guessing" ) record = TurnRecord( turn_id=commit.turn_id, committed_attempt_id=commit.committed_attempt_id, ) for v in verdicts: if v.turn_id != commit.turn_id: raise MisattributedVerdictError( f"verdict for turn {v.turn_id!r} offered to join for {commit.turn_id!r}" ) if v.attempt_id is None or v.content_hash is None: # Unpinned verdict: cannot prove which attempt it judged. record.quarantined.append(v) continue if v.attempt_id != commit.committed_attempt_id: # Verdict about a non-committed attempt (retry shadow / draft). record.quarantined.append(v) continue if v.content_hash != commit.committed_content_hash: # Same attempt id, different content: hash mismatch is fatal. raise MisattributedVerdictError( f"content hash mismatch on committed attempt {v.attempt_id}: " f"{v.content_hash!r} != {commit.committed_content_hash!r}" ) if v.scope == "window": # Window verdicts are retained but do not roll into turn severity # without an explicit declared rollup policy. record.quarantined.append(v) continue record.verdicts.append(v) return record ```