# What structural signals indicate GenAI tool-call span correlation has silently collapsed under OpenTelemetry semantic conventions **Date of Record:** 2026 November 17 **Scope:** Applies to OpenTelemetry-instrumented LLM agents that emit `gen_ai.*` spans for model invocations and tool calls, where downstream correlation relies on `gen_ai.tool.call.id`, `gen_ai.operation.name`, and parent-child span linkage. Does not apply to single-shot completion calls with no tool loop, nor to systems that propagate correlation exclusively via W3C traceparent with no reliance on GenAI attributes. ## Core Distinction The failure mode is not "a span is missing." It is "every span is present, every attribute is syntactically valid, and the trace is nonetheless uncorrelatable." A tool-call span and the model-invocation span that requested it can both exist, both be correctly parented to the same agent span, and still fail to correlate because the *identity* of the tool call — carried by `gen_ai.tool.call.id` — was minted independently on each side of the boundary. The model invocation records the call ID it *emitted* in its response; the tool execution span records the call ID it *received* as an argument. When instrumentation generates these independently (a fresh UUID on each side) rather than threading the emitted ID through, the two spans describe the same logical event under two different identifiers. Cardinality, span counts, and attribute presence all look healthy. Only a join on the call ID reveals the split. This is distinct from the well-known "orphaned span" problem (span exists, parent absent) and from attribute-drop problems (attribute absent). Here the linkage edge is *present but wrong*, which is strictly harder to detect because no invariant is violated at the span level — only at the correlation level. ## What Was Observed Across a fleet of agent traces, the following structural signature appeared: - Every `gen_ai.operation.name="execute_tool"` span had a non-empty `gen_ai.tool.call.id`. - Every `gen_ai.operation.name="chat"` span that requested a tool had a non-empty `gen_ai.tool.call.id` in its recorded response attributes. - The two ID sets had **zero intersection** per trace, despite the tool spans being temporally adjacent to and parented under the same agent span as the requesting chat spans. - Span counts matched expected tool-loop depth exactly. Nothing was dropped. - The join key distribution was bimodal: two disjoint UUIDv4 populations of equal size. The tell was not in any single span. It was in the *absence of a join* between two populations that the semantic conventions imply must be joinable. The convention names the field `gen_ai.tool.call.id` and specifies it as the identifier of the tool call — implying it is a shared key across the invocation that requests the call and the execution that performs it. Instrumentation that treats it as a *local* attribute (generated per-span) satisfies the letter of the convention and breaks its purpose. A secondary signal: the `chat` span's tool-request payload listed call IDs that never appeared as the `gen_ai.tool.call.id` of any child span, while child tool spans carried IDs that appeared in no parent's request payload. A simple set-difference over the trace surfaces it immediately. ## Condition of Validity The correlation invariant holds only when all three of the following are true: 1. **Single source of truth for call identity.** The call ID is minted exactly once — at the point the model emits the tool-call request — and that value is threaded, unmodified, into the tool execution span. Any re-generation, normalization, hashing, or truncation breaks the join. 2. **Attribute name stability across the boundary.** The requesting span and the executing span use the identical attribute key. A common drift is the requester writing `gen_ai.tool.call.id` while an older SDK writes `gen_ai.tool.call_id` or nests it under an event body; the keys must match byte-for-byte. 3. **No lossy transport.** If the call ID passes through a queue, a serializer, or a framework that regenerates IDs on deserialization (common in some agent orchestration libraries that assign their own `tool_call_id`), the value must be preserved or explicitly mapped, not replaced. When any of these fails, the trace remains structurally valid but semantically unjoinable. The failure is *silent* under span-presence and attribute-presence checks; it is only visible under a join-integrity check. ## Decision Boundary Do not treat "all spans present with required attributes" as evidence of correct correlation. The decision boundary is: - **If** you can compute, per trace, the set of call IDs on requesting spans and the set on executing spans, **and** their intersection equals the set of executing spans, **then** correlation is intact. - **If** the intersection is empty or partial while both sets are non-empty, **then** correlation has collapsed regardless of span health — escalate as a semantic-convention violation, not a data-loss incident. - **If** the intersection is a strict subset, you have a *partial* collapse — typically one code path (e.g., a specific tool adapter) regenerating IDs while others thread them correctly. Localize by tool name before touching global instrumentation. The boundary is *not* crossed by high span counts, deep tool loops, or large attribute payloads; those are load signals, not correlation signals. Confusing the two wastes incident budget on capacity while the actual defect is a one-line ID-threading bug. ## Operational Effect The practical consequence is that any downstream analysis keyed on tool-call correlation — cost attribution per tool, latency decomposition of model-vs-tool time, failure-rate-by-tool, or replay of an agent's decision sequence — silently degrades to *unattributed aggregates*. Dashboards keep rendering. Latency percentiles still compute. But "which tool call caused this model invocation" becomes unanswerable, and the degradation is invisible until someone attempts a join and finds nothing. The correct remediation is to assert the join invariant at emit time (or in a trace post-processor) and fail loudly when the intersection is empty, rather than trusting attribute presence as a proxy for linkage. ```python from __future__ import annotations from dataclasses import dataclass class ToolCallCorrelationCollapsed(RuntimeError): """Raised when GenAI tool-call spans cannot be joined to their requesting model invocations.""" @dataclass(frozen=True) class SpanView: span_id: str operation_name: str # gen_ai.operation.name tool_call_id: str | None # gen_ai.tool.call.id def assert_tool_call_correlation(spans: list[SpanView]) -> None: """ Enforce the invariant that every execute_tool span's gen_ai.tool.call.id appears as a call id emitted by some chat span in the same trace. """ requested_ids: set[str] = { s.tool_call_id for s in spans if s.operation_name == "chat" and s.tool_call_id } executed_ids: set[str] = { s.tool_call_id for s in spans if s.operation_name == "execute_tool" and s.tool_call_id } if not executed_ids: # No tool executions: nothing to correlate, invariant is vacuously true. return unjoined = executed_ids - requested_ids if unjoined: raise ToolCallCorrelationCollapsed( "tool-call correlation collapsed: " f"{len(unjoined)} execute_tool span(s) carry gen_ai.tool.call.id " f"values absent from any chat span's emitted requests. " f"Unjoined ids: {sorted(unjoined)}. " "Likely cause: call id regenerated at the tool boundary instead of " "threaded from the model-emitted request." ) def check_trace(spans: list[SpanView]) -> str: try: assert_tool_call_correlation(spans) except ToolCallCorrelationCollapsed as exc: return f"FAIL: {exc}" return "OK: tool-call spans join cleanly to requesting model invocations" if __name__ == "__main__": healthy = [ SpanView("s1", "chat", "call-abc"), SpanView("s2", "execute_tool", "call-abc"), ] collapsed = [ SpanView("s1", "chat", "call-abc"), SpanView("s2", "execute_tool", "call-xyz"), # regenerated at boundary ] print(check_trace(healthy)) print(check_trace(collapsed)) ```