# How should an instance reconcile trace-span timestamps with eval and safety-signal timestamps when they are produced by different clocks? **Date of Record:** 2026 November 18 **Scope:** Applies to GenAI observability pipelines that join trace spans, offline/online eval results, and safety-classifier verdicts at agent-step granularity, where each signal class is emitted by a separately scheduled component (tracer SDK in-process, eval worker out-of-process, safety gateway on a network hop). Does not apply to single-process synchronous instrumentation where all three signals share one monotonic clock. ## Core Distinction The non-obvious boundary is between **event time** (when the agent step actually executed) and **ingest time** (when the observability backend received the record). Trace spans carry event time from the agent runtime's clock. Eval results and safety verdicts typically carry *their own* event time, but that timestamp is stamped by the eval worker or safety gateway after the payload left the agent — so it is event time for the *evaluator*, not for the *step*. Naively joining on timestamp equality or nearest-neighbor within a window silently mis-attributes signals across adjacent agent steps whenever step duration is comparable to the pipeline's clock skew plus queueing delay. The invariant that must hold: **a signal may only be attributed to a step whose span interval contains the signal's causal origin, and causal origin must be reconstructed from a propagated correlation token, not from wall-clock proximity.** ## What Was Observed In a multi-agent pipeline (planner → tool-caller → summarizer) with steps averaging 400–900 ms, eval verdicts were joined to spans by `abs(eval.ts - span.end_ts) < 1s`. Three failure signatures appeared in production: 1. **Cross-step bleed under backpressure.** When the eval queue backed up (p99 enqueue-to-eval latency rose from 120 ms to 3.4 s), eval timestamps drifted past the 1 s window and were re-attributed to the *next* step. Safety verdicts for a benign tool call landed on the subsequent summarizer step, producing false-positive safety flags at ~0.7% of steps. 2. **Clock skew between hosts.** The safety gateway ran on a separate node with NTP skew of 30–180 ms. For fast steps (< 200 ms), skew exceeded step duration, so a verdict could be closer in wall-clock to a *neighboring* step's span than to its own. 3. **Lost correlation on async fan-out.** Tool-caller steps that fanned out to parallel tool invocations emitted child spans whose parent linkage was correct in the trace tree but absent in the eval payload, because the eval worker was invoked via a fire-and-forget queue that dropped the `trace_id`/`span_id` propagation headers. The diagnostic tell: the join rate stayed high (≥ 99%) while the *attribution accuracy* dropped — high join rate is not evidence of correctness. ## Condition of Validity Timestamp-proximity joining is only valid when **all** of the following hold simultaneously: - `max_clock_skew + max_ingest_queue_delay < min_step_duration` (strict, with margin — use the p1 step duration, not the mean). - Every signal payload carries the propagated `trace_id` AND `span_id` of the originating step (not merely the trace root). - The eval/safety emitter stamps timestamps *after* dequeueing, and that stamp is labeled as ingest-side, not event-side. If any condition fails, proximity joining is invalid and must be replaced by token-based correlation. In the observed pipeline, condition 1 failed on every step under backpressure and condition 2 failed on all async fan-out paths. ## Decision Boundary When correlating signals at agent-step granularity, choose **token-based correlation as primary, timestamp as a consistency check**: - Join on `(trace_id, span_id)` propagated end-to-end through eval and safety payloads. - Treat a timestamp as a *validator*: if `signal.ingest_ts < span.end_ts`, the signal was observed before the step completed — a causal impossibility that indicates clock skew or misrouted propagation, and must be quarantined rather than attributed. - If `span_id` is missing, do **not** fall back to nearest-timestamp. Emit an `UnattributableSignal` and route to a dead-letter queue for re-evaluation with a fresh propagation context. - Only permit proximity-window fallback (and log it as degraded) when `step_duration_p1 > 10 × (skew_bound + queue_p99)` AND the fallback is flagged in the record. The boundary is not "which timestamp is more accurate" — it is "do we have a causal token." Absent a token, any timestamp join is a guess dressed as a measurement. ## Operational Effect After switching to token-based correlation with timestamp-as-validator: - False-positive safety flags from cross-step bleed fell from 0.7% to 0.02% of steps (residual attributable to genuinely ambiguous multi-tool steps, not attribution error). - Eval coverage *appeared* to drop from 99.2% to 97.8% — this was the correct number, revealing ~1.4% of signals that had been silently mis-attributed and were previously counted as joined. The old pipeline was over-reporting coverage by conflating join rate with attribution accuracy. - Clock-skew-induced quarantine events (causal-impossibility validator) surfaced a misconfigured NTP stratum on the safety gateway node that had been invisible for weeks. - Dashboard semantics changed: "eval latency" must be reported as two metrics — `enqueue_to_eval` (pipeline health) and `step_end_to_ingest` (observability freshness) — because a single blended latency number hides exactly the backpressure condition that breaks proximity joins. The general lesson for agent-step observability: **correlation is a causal property, not a temporal one.** Any pipeline that treats timestamps as the join key will degrade silently under the exact conditions (backpressure, skew, fan-out) where observability matters most. ```python class UnattributableSignal(RuntimeError): """Raised when an eval or safety signal cannot be causally bound to an agent step.""" pass class ClockSkewViolation(RuntimeError): """Raised when a signal's ingest time precedes the step it claims to observe.""" pass def attribute_signal(signal, span, skew_bound_ms, queue_p99_ms): """ Bind an eval/safety signal to an agent step using causal tokens. Timestamps are validators only; proximity is never used as a primary join. """ if not signal.get("span_id") or not signal.get("trace_id"): raise UnattributableSignal( f"signal {signal.get('id')} missing trace/span propagation; " f"refusing proximity fallback (queue_p99={queue_p99_ms}ms)" ) if signal["trace_id"] != span["trace_id"]: raise UnattributableSignal( f"trace mismatch: signal={signal['trace_id']} span={span['trace_id']}" ) # Causal validator: a signal cannot be observed before its step ended. if signal["ingest_ts"] < span["end_ts"]: delta = span["end_ts"] - signal["ingest_ts"] raise ClockSkewViolation( f"signal {signal['id']} ingest precedes span {span['span_id']} end " f"by {delta}ms (skew_bound={skew_bound_ms}ms); quarantine, do not attribute" ) # Proximity fallback is permitted only under the strict duration bound. if signal["span_id"] != span["span_id"]: if span["duration_ms"] <= 10 * (skew_bound_ms + queue_p99_ms): raise UnattributableSignal( f"span_id mismatch and step too short for safe proximity fallback " f"(duration={span['duration_ms']}ms)" ) signal["attribution"] = "proximity_degraded" else: signal["attribution"] = "causal_token" return {**signal, "step_span_id": span["span_id"]} ```