# Under what conditions must an instance treat a retried span as a new causal root rather than a continuation of the original trace **Date of Record:** 2026 November 13 **Scope:** Applies to agentic systems that emit OpenTelemetry-style spans across retries, delegated sub-agents, and asynchronous tool callbacks where the parent span may have already been closed (or its process terminated) before the child completes. Does not apply to purely synchronous single-process agent loops where the entire causal chain lives within one uninterrupted span lifetime. ## Core Distinction The distinction that matters operationally is between **causal identity** (the logical unit of work the user asked for) and **span identity** (the physical record emitted by a runtime). Retries, sub-agent delegation, and async tool callbacks all violate the naive assumption that one trace equals one logical operation. The non-obvious boundary is this: a retried span is a *continuation* of the original trace only if the retry is causally downstream of the same initiating request **and** the trace context was propagated through the retry mechanism itself. If the retry is triggered by a supervisor, a queue redelivery, or a model-driven re-plan that has no access to the original `traceparent`, the retry is a *new causal root* that must be stitched to the original by a correlation key — not by span parentage. The failure mode is subtle because both choices "work" in a dashboard. Continuation-by-default produces traces that look complete but attribute retry latency and failure cost to the wrong parent, inflating sub-agent p99 and hiding the true retry amplification factor. Root-by-default produces disconnected traces that lose the user-request lineage entirely. Neither is universally correct; the condition of validity determines which. ## What Was Observed In a production agentic loop (planner → N sub-agents → async tool callbacks), three concrete anomalies appeared: 1. **Retry amplification was invisible.** A tool call failed with a transient 503 and was retried three times by the agent's own retry policy. All three attempts were emitted as children of the same `tool_call` span because the retry loop reused the in-process `SpanContext`. Aggregate trace latency reported the *last* attempt's duration, not the sum, because the exporter deduplicated by span ID on the retry path where the SDK reused IDs. The true cost was 4× the reported figure. 2. **Sub-agent spans orphaned on supervisor restart.** A sub-agent was dispatched via a message queue. The supervisor crashed and was replaced; the new supervisor had no `SpanContext` for the in-flight work, so the sub-agent's spans were emitted with a fresh root trace ID. The user request appeared as two unrelated traces, and the sub-agent's failure was never correlated to the originating request. 3. **Async tool callback attached to a dead parent.** A long-running tool (webhook-based) completed after the parent agent process had exited. The callback handler had the parent `span_id` but the parent span was already closed and its process gone. The exporter accepted the child span but the backend dropped it as an orphan (parent not found within the retention window), so the most expensive operation in the trace — the async completion — was silently lost. In all three cases the system "worked" and the traces "looked fine" until someone tried to answer: *what did this one user request actually cost, and where did it fail?* ## Condition of Validity A retried span is a valid **continuation** of the original trace if and only if all of the following hold: - The retry is initiated within the same causal process tree as the original attempt (the retry policy holds the original `SpanContext` in memory). - The span is not emitted with a reused `span_id`; each attempt gets a fresh `span_id` and an explicit `retry.attempt` attribute plus a link to the prior attempt. - The parent span has not been closed and its process has not exited before the child span is exported, OR the exporter is configured with span links (not parent references) so the backend can resolve the relationship out-of-band. If any of these fail — queue redelivery, supervisor replacement, process exit before async completion, or model-driven re-planning without context propagation — the retried span **must be treated as a new causal root** and stitched to the original via a stable correlation key (e.g., `request.id`, `conversation.id`, or a `trace.group` attribute), with an explicit span link to the original root when available. The invariant: **span parentage is a within-process-tree relationship; cross-process-tree causality must be expressed as a link plus a correlation attribute, never as a parent reference to a span that may not exist at export time.** ## Decision Boundary The boundary is crossed at the moment the retry/sub-agent/callback loses access to a *live* parent `SpanContext`. Concretely: - **Same process, same in-memory context, parent span open** → continuation. Emit child with parent reference, fresh span ID, `retry.attempt` counter. - **Same process, but parent span already ended** (e.g., async callback after planner returned) → new root with correlation attribute + span link to the ended parent. Do not use a parent reference; the backend may not resolve it. - **Different process** (queue redelivery, supervisor restart, separate sub-agent worker) → new root. Propagate the correlation key through the message payload; propagate `traceparent` only if the receiving process will emit within the parent's lifetime, otherwise emit a link. - **Model-driven re-plan** (the agent decides to retry with a different strategy) → new root. This is a new logical attempt, not a transport-level retry; conflating them hides re-planning cost. The decision is not about whether the operation is "the same work." It is about whether the causal parent is *resolvable at export time*. If it is not, parentage is a lie that the backend will either drop or mis-attribute. ## Operational Effect Applying this boundary correctly changes three operational signals: 1. **Retry amplification becomes measurable.** Each attempt is a distinct span with a link to its predecessor, so `count(attempt) / count(root)` is the true amplification factor, and summed latency reflects real cost rather than last-attempt duration. 2. **Orphan rate drops to near zero.** Async callbacks and replaced supervisors emit resolvable roots with correlation attributes, so the backend no longer drops spans for missing parents. The residual orphan rate becomes a *signal* (context propagation bug) rather than noise. 3. **Attribution is honest.** Sub-agent latency is attributed to the sub-agent's own root, with a link back to the planner, so a slow sub-agent no longer inflates the planner's p99 — and a planner that over-delegates shows up as fan-out, not as slow children. The cost is that traces are no longer single tree structures; they are DAGs assembled by the query layer from links and correlation keys. This is the correct mental model for agentic systems, where the causal graph is genuinely a DAG, not a tree. The failure mode this prevents is the most expensive one in agent observability: a trace that looks complete and is silently wrong. ```python from __future__ import annotations from dataclasses import dataclass, field from enum import Enum from typing import Optional class CausalStitchingError(RuntimeError): """Raised when a retried/sub-agent/async span cannot be validly stitched.""" class StitchMode(Enum): CONTINUATION = "continuation" # parent reference, live parent in same process tree NEW_ROOT_LINKED = "new_root_linked" # new root + span link + correlation key @dataclass class SpanContext: trace_id: str span_id: str process_id: str is_open: bool correlation_key: Optional[str] = None @dataclass class RetryAttempt: original: SpanContext retry_process_id: str retry_has_live_parent: bool correlation_key: Optional[str] = None attempt_number: int = 1 links: list[str] = field(default_factory=list) def decide_stitch_mode(attempt: RetryAttempt) -> StitchMode: """ Enforce the invariant: span parentage requires a live, resolvable parent in the same process tree. Otherwise the retry is a new causal root that must be stitched via correlation key + span link. """ same_process = attempt.retry_process_id == attempt.original.process_id parent_resolvable = attempt.original.is_open and same_process if parent_resolvable and attempt.retry_has_live_parent: return StitchMode.CONTINUATION # Boundary crossed: parent is not resolvable at export time. if not attempt.correlation_key: raise CausalStitchingError( f"Retry attempt {attempt.attempt_number} of span " f"{attempt.original.span_id} (trace {attempt.original.trace_id}) " f"crossed the causal boundary (same_process={same_process}, " f"parent_open={attempt.original.is_open}) but carries no " f"correlation_key. Cannot stitch without a stable key; refusing " f"to emit a parent reference to an unresolvable span." ) return StitchMode.NEW_ROOT_LINKED def emit_retry_span(attempt: RetryAttempt) -> dict: mode = decide_stitch_mode(attempt) if mode is StitchMode.CONTINUATION: return { "trace_id": attempt.original.trace_id, "parent_span_id": attempt.original.span_id, "retry.attempt": attempt.attempt_number, "links": attempt.links, } return { "trace_id": f"root-{attempt.correlation_key}", "parent_span_id": None, "correlation_key": attempt.correlation_key, "links": [attempt.original.span_id, *attempt.links], "retry.attempt": attempt.attempt_number, "causal.root": True, } ```