# In what scenarios must an instance-level GenAI span be split before OpenTelemetry export: the multi-step agent trace invariant failure **Date of Record:** 2026 November 16 **Scope:** Applies to OpenTelemetry GenAI semantic conventions (`gen_ai.*`) when a single logical agent invocation emits multiple model calls, tool calls, or nested sub-agent runs that are represented as spans under one trace. Does not apply to single-shot inference instrumentation, nor to non-GenAI spans that merely carry a `gen_ai.request.model` attribute for routing metadata. ## Core Distinction The GenAI semantic conventions define span-level attributes (`gen_ai.operation.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.response.finish_reasons`) as belonging to *one operation*. The non-obvious boundary is this: a span may be **structurally a parent** (it has child spans for tool calls and sub-agent steps) while **semantically it is an aggregate** (it summarizes the whole agent run). The conventions do not define an aggregate span type. When an implementer attaches summed token usage and a single `finish_reasons` array to an agent-orchestrator span that also has children, the resulting span is neither a valid operation span nor a valid aggregate — and downstream consumers that assume operation-span semantics will silently double-count or misattribute cost. The distinction that matters operationally: **a span is a unit of attribution, not a unit of narration.** If a span's attributes describe more than one model call, it is not a GenAI operation span and must not be exported as one. ## What Was Observed In a multi-step agent (planner → tool executor → synthesizer, all under one orchestrator span), the orchestrator span was emitted with: - `gen_ai.operation.name = "chat"` - `gen_ai.request.model = "claude-3-5-sonnet"` (the planner's model) - `gen_ai.usage.input_tokens` = sum across all three steps - `gen_ai.response.finish_reasons = ["tool_use", "tool_use", "end_turn"]` Three failure modes surfaced at the collector/backend boundary: 1. **Cost double-counting.** Backends that compute cost as `usage × model_price` on every span carrying `gen_ai.usage.*` billed the orchestrator's summed tokens *in addition to* the child spans' individual usage. Observed inflation: 2.1× on a three-step run (orchestrator + three children), because the orchestrator's sum equals the children's sum. 2. **`finish_reasons` cardinality violation.** The convention models `finish_reasons` as a list aligned to a single response (one entry per candidate, typically `["end_turn"]`). Emitting `["tool_use", "tool_use", "end_turn"]` on one span produced dashboards that counted three "responses" per orchestrator, breaking per-response latency and refusal-rate panels. 3. **Model misattribution.** The synthesizer's output tokens were billed at the planner's model price because the orchestrator declared a single `gen_ai.request.model`. In mixed-model agents (cheap planner, expensive synthesizer) this understated or overstated cost by 30–60% depending on direction. None of these produced an error. All produced *plausible, wrong* telemetry — the worst class of observability failure. ## Condition of Validity The invariant holds as stated **only when** the following are all true: - The backend or processor aggregates `gen_ai.usage.*` per span without a parent/child deduplication pass. (If the backend deduplicates by summing only leaf spans, the double-count vanishes but misattribution remains.) - The orchestrator span's `gen_ai.request.model` differs from at least one child's model, **or** the orchestrator carries usage that is the sum of children. If the orchestrator carries *no* usage attributes and *no* model attribute — acting purely as a structural parent — it is a legitimate span and must be kept. - The trace is consumed by cost, SLO, or eval pipelines that treat `gen_ai.*` attributes as authoritative per-operation facts. The invariant does **not** hold for backends that explicitly implement aggregate-span handling (rare, and never portable across vendors). Do not assume portability. ## Decision Boundary Apply this test at instrumentation time, before export: 1. Does the span represent exactly one model call or one tool call? → Emit as a GenAI operation span with usage and model attributes. Keep. 2. Does the span represent an orchestration step whose usage equals the sum of its GenAI children? → **Strip `gen_ai.usage.*`, `gen_ai.request.model`, and `gen_ai.response.finish_reasons` from the orchestrator.** Keep the span as a structural parent with only `gen_ai.operation.name = "invoke_agent"` (or your vendor's agent operation) and non-usage attributes. Let children carry the facts. 3. Does the span represent an orchestration step with usage *not* attributable to children (e.g., a hidden routing call)? → Split: emit a child span for the routing call, keep the parent structural. The boundary is crossed the moment a span's `gen_ai.usage.input_tokens` is not the token count of the operation the span *names*. At that point the span has become an aggregate wearing an operation span's clothes. ## Operational Effect After enforcing the split, the observed effects on a production agent fleet (three-step, mixed-model, ~40k runs/day): - Cost attribution error dropped from 2.1× inflation to within 3% of provider invoices. - `finish_reasons`-derived response counts aligned with actual model calls; refusal-rate panels stabilized. - Trace size grew ~8% (extra structural spans, fewer fat orchestrator spans), which was acceptable against the 40% reduction in backend aggregation CPU from removing redundant usage sums. - One regression: dashboards that had been reading the orchestrator's summed tokens as a convenience "total tokens per agent run" broke and had to be rewritten to sum leaf spans. This is the correct migration direction; document it before rollout. The general lesson: **do not let a span narrate more than it can be attributed to.** Structural parent spans are cheap; misattributed aggregates are expensive and silent. ```python class GenAISpanModelBoundaryError(RuntimeError): """Raised when a span carries GenAI usage/model attributes that do not describe the single operation the span names. Exporting such a span corrupts cost attribution and response-cardinality metrics downstream.""" pass GENAI_USAGE_ATTRS = ( "gen_ai.usage.input_tokens", "gen_ai.usage.output_tokens", "gen_ai.request.model", "gen_ai.response.finish_reasons", ) def validate_genai_span(span_name: str, attributes: dict, child_genai_usage_sums: dict | None) -> None: """Enforce the attribution invariant before export. A span may carry GenAI usage attributes only if those attributes describe the single operation the span names. If the span has GenAI children whose summed usage equals this span's usage, the span is an aggregate and must be stripped to a structural parent. """ present = {k: attributes[k] for k in GENAI_USAGE_ATTRS if k in attributes} if not present: return # structural parent or non-GenAI span: fine if child_genai_usage_sums is None: return # leaf operation span: usage is authoritative span_input = present.get("gen_ai.usage.input_tokens") child_input = child_genai_usage_sums.get("gen_ai.usage.input_tokens") if span_input is not None and child_input is not None and span_input == child_input: raise GenAISpanModelBoundaryError( f"span {span_name!r} carries gen_ai.usage.input_tokens={span_input} " f"equal to the sum of its GenAI children ({child_input}). " "This span is an aggregate, not an operation span. Strip " "gen_ai.usage.*, gen_ai.request.model, and gen_ai.response.finish_reasons " "and re-export as a structural parent; let children carry the facts." ) if "gen_ai.response.finish_reasons" in present: reasons = present["gen_ai.response.finish_reasons"] if isinstance(reasons, (list, tuple)) and len(reasons) > 1: raise GenAISpanModelBoundaryError( f"span {span_name!r} has finish_reasons={reasons!r} with cardinality " f"{len(reasons)}. finish_reasons models one response's candidates, " "not a sequence of steps. Split into per-step child spans." ) def export_span(span_name: str, attributes: dict, child_genai_usage_sums: dict | None, exporter) -> None: validate_genai_span(span_name, attributes, child_genai_usage_sums) exporter(span_name, attributes) ```