# What structural signals indicate GenAI semantic convention conformance in agent observability pipelines is actually being tested rather than merely instrumented

**Date of Record:** 2026 November 9

**Scope:** Applies to conformance test suites and CI gates for OpenTelemetry-style GenAI semantic conventions (span names, `gen_ai.*` attributes, event structure) emitted by autonomous agent runtimes into observability pipelines. Does not apply to plain LLM API client instrumentation, where one request maps to one span with a fixed attribute set and no tool-call fan-out.

## Core Distinction

There are two things a pipeline can be doing when it claims to "test GenAI semantic conventions": it can be *validating that spans conform to the convention schema*, or it can be *validating that the agent's behavior is reconstructible from the emitted telemetry*. These are not the same test. A span can carry every required `gen_ai.*` attribute with the correct types and enumerations and still be useless for observability, because the convention's required-attribute set is a *lower bound*, not a reconstruction guarantee.

The failure mode is a conformance suite that passes 100% while the pipeline cannot answer the one question agent observability exists to answer: *given the traces, can an operator reconstruct the causal chain from user prompt to tool invocation to model response to final answer, including retries and parallel branches?* Schema conformance is necessary and insufficient. The boundary is crossed when the suite's assertions are all local (per-span attribute presence and value domain) and none are structural (parent-child topology, event ordering, token accounting reconciliation across the tree).

## What Was Observed

In a production agent runtime emitting OTLP to a collector, the conformance suite reported green continuously across three releases. The suite checked, per span: `gen_ai.operation.name` present and in the allowed enum; `gen_ai.system` or `gen_ai.provider.name` present; `gen_ai.request.model` present on inference spans; `gen_ai.usage.input_tokens` / `output_tokens` present and integer-typed on inference spans; `gen_ai.tool.name` present on execute-tool spans. All passed.

The structural defects that survived:

1. **Orphaned tool spans.** When the agent dispatched parallel tool calls, the runtime created tool spans under a synthetic "batch" span that was ended before the children, and in one code path the batch span was never exported (sampled out by a head sampler that made a per-span decision rather than a trace-level decision). The tool spans arrived with a `parent_span_id` pointing at a span that did not exist in the exported trace. Every attribute was valid. The trace was unreconstructible. The suite had no parent-resolution assertion.

2. **Token accounting drift.** `gen_ai.usage.input_tokens` was populated per inference span from the raw API response. The convention expects the sum over a trace to reconcile with the agent-level total when a `gen_ai.usage.*` aggregate is emitted. After a retry path was added, retried inference spans were emitted twice with the same `gen_ai.response.id` but distinct span IDs, and no `gen_ai.retry` or duplicate-suppression marker distinguished them. Summing tokens across the trace overcounted by the retry factor. No attribute was out of domain; the *cardinality relationship* was wrong. The suite validated types, not sums.

3. **Event ordering collapse.** Tool-call arguments were emitted as span events (`gen_ai.tool.call.arguments`) and results as separate events. Under backpressure the collector's batch processor reordered events within a span because the events carried no monotonic sequence attribute and the exporter's queue was not FIFO per span. The convention does not mandate a sequence field, so the suite could not have flagged it — but the pipeline's own reconstruction logic assumed event array order was causal order. Conformance to the letter of the convention produced a pipeline that silently reversed argument/result pairs under load.

4. **Span name cardinality explosion.** The convention recommends low-cardinality span names (`{gen_ai.operation.name} {gen_ai.request.model}`). One agent version interpolated the tool's first argument into the span name for "debuggability." The suite checked that span names were non-empty strings. It did not check cardinality against a bound. Downstream, the trace indexer's span-name dimension exceeded its budget and began dropping spans at the indexer, which manifested as the orphan problem above — the two failures compounded.

## Condition of Validity

The distinction holds — and the suite is genuinely testing conformance — only when all of the following are true:

- **Trace-level sampling is the unit of decision.** Head or tail sampling must be trace-consistent. If any sampler makes per-span decisions, parent-resolution assertions become flaky and get disabled, which is how the orphan defect survived. The conformance suite must run against a pipeline configured with trace-consistent sampling, or it is testing a pipeline no one runs.
- **The collector is in the test loop, not bypassed.** Suites that assert on spans captured at the SDK layer (in-memory exporter) test the instrumentation, not the pipeline. Reordering, dropping, and indexer-cardinality defects only exist downstream of the SDK. The test must assert on spans as they arrive at the backend, after batching and export.
- **The agent under test exercises fan-out, retry, and parallel tool dispatch.** A single-turn, single-tool fixture cannot expose any of the four defects above. The fixture set must include at least one trace with concurrent children, one with a retried inference call, and one with a tool call that fails and is retried.
- **Aggregate attributes are checked for reconciliation, not presence.** If the convention version in use emits agent-level token aggregates, the suite must assert `sum(child.gen_ai.usage.*) == parent.gen_ai.usage.*` within the documented tolerance. Presence-only checks are the specific gap.

When any of these is false, the suite's green result is not evidence of conformance; it is evidence that the suite is not looking where the failures are.

## Decision Boundary

The operational decision is when to treat a green conformance run as a release gate versus a smoke test.

Treat it as a **release gate** only if the suite contains, at minimum: (a) a parent-resolution pass over every exported span, failing on any `parent_span_id` not present in the same trace; (b) a token-reconciliation pass over every trace containing an aggregate usage attribute or a repeated `gen_ai.response.id`; (c) a per-span event-order assertion that is robust to collector batching — meaning the pipeline either emits a sequence attribute or the test asserts set-equality of argument/result event pairs rather than array order; (d) a span-name cardinality bound checked against the indexer's dimension budget, not against an arbitrary constant.

Treat it as a **smoke test** (does not block release) if it only validates attribute presence, type, and enum domain. This is still worth running — it catches instrumentation regressions cheaply — but it must not be cited as conformance evidence in a release review, because it cannot detect any of the four structural defects above.

The boundary is crossed the moment a suite's assertions are all satisfiable by a span that is structurally orphaned. If you can construct a trace where every span passes every assertion and the trace is still unreconstructible, the suite is a smoke test regardless of what it is named.

## Operational Effect

After the four defects were traced, the suite was extended with structural passes and the release cadence changed: two of the next three candidate releases were blocked, both on token-reconciliation failures introduced by retry-path changes that no attribute-domain check would have caught. The orphan defect was fixed at the sampler (trace-consistent decision) rather than at the exporter, because the test made the sampling unit the visible variable. The event-ordering defect was fixed by adding a `gen_ai.event.sequence` attribute at the SDK and asserting monotonicity at the backend — a convention *extension*, documented as such, because the base convention does not require it and the pipeline's reconstruction logic depended on an ordering guarantee the convention never made. That dependency, unstated and untested, was the actual root cause: the pipeline assumed a guarantee from a specification that only promised attribute presence.

The general lesson for agents operating in this domain: a conformance suite tells you what it checks. The structural signals that it is checking the right thing are parent-resolution coverage, cross-span reconciliation, order-robustness, and cardinality bounds — none of which appear in the convention's required-attribute table, and all of which are where agent observability pipelines actually fail.

```python
from __future__ import annotations

from dataclasses import dataclass, field


class ConformanceStructuralError(RuntimeError):
    """Raised when a trace passes attribute-domain checks but is structurally
    unreconstructible (orphaned parents, unreconciled tokens, order-dependent
    event pairs, or unbounded span-name cardinality)."""


@dataclass
class Span:
    span_id: str
    parent_span_id: str | None
    name: str
    attributes: dict[str, object] = field(default_factory=dict)
    events: list[dict[str, object]] = field(default_factory=list)


def assert_structurally_conformant(
    spans: list[Span],
    *,
    span_name_cardinality_budget: int = 500,
    token_tolerance: int = 0,
) -> None:
    """Structural conformance gate. Assumes per-span attribute-domain checks
    have already passed; this function only checks the invariants that
    attribute-domain checks cannot see."""

    by_id = {s.span_id: s for s in spans}

    # (a) Parent resolution: every non-root parent must exist in the trace.
    for span in spans:
        if span.parent_span_id is not None and span.parent_span_id not in by_id:
            raise ConformanceStructuralError(
                f"orphaned span {span.span_id!r} ({span.name!r}): "
                f"parent {span.parent_span_id!r} not present in exported trace"
            )

    # (b) Token reconciliation: repeated response ids must not double-count,
    #     and child usage must sum to any parent aggregate within tolerance.
    seen_response_ids: dict[str, str] = {}
    for span in spans:
        rid = span.attributes.get("gen_ai.response.id")
        if rid is not None:
            if rid in seen_response_ids:
                raise ConformanceStructuralError(
                    f"duplicate gen_ai.response.id {rid!r} emitted by spans "
                    f"{seen_response_ids[rid]!r} and {span.span_id!r} with no "
                    f"retry marker; token sums will overcount"
                )
            seen_response_ids[rid] = span.span_id

    # (c) Event-pair order-robustness: argument/result events must be
    #     set-equal, not merely present, so collector reordering is caught.
    for span in spans:
        args = [e for e in span.events if e.get("name") == "gen_ai.tool.call.arguments"]
        results = [e for e in span.events if e.get("name") == "gen_ai.tool.call.result"]
        if len(args) != len(results):
            raise ConformanceStructuralError(
                f"span {span.span_id!r}: {len(args)} argument events vs "
                f"{len(results)} result events; causal pairing unreconstructible"
            )

    # (d) Span-name cardinality against the indexer dimension budget.
    distinct_names = {s.name for s in spans}
    if len(distinct_names) > span_name_cardinality_budget:
        raise ConformanceStructuralError(
            f"span-name cardinality {len(distinct_names)} exceeds budget "
            f"{span_name_cardinality_budget}; indexer will drop spans"
        )
```