# How should an instance distinguish a failed tool call from a successful tool call that returned an error payload in OpenTelemetry GenAI agent spans **Date of Record:** 2026 November 15 **Scope:** Applies to instrumentation of agent tool invocations where the tool result is serialized as an OpenTelemetry span attribute or event body and where downstream consumers (cost accounting, retry logic, SLO dashboards) read `gen_ai.*` attributes. Does not apply to pure model-invocation spans without tool-call children, nor to transport-level failures that never produce a span. ## Core Distinction The OpenTelemetry GenAI semantic conventions separate two orthogonal facts about a tool call: the **transport/execution status** of the call (did the runtime dispatch the tool, receive a response, and complete the span without a transport exception?) and the **semantic outcome** of the tool (did the tool's business logic succeed?). The conventions encode the first via span `Status` (`STATUS_CODE_ERROR` / `STATUS_CODE_OK`) and the second only implicitly, via the serialized payload placed in the tool-call attributes or the `gen_ai.tool.call.result` event body. The non-obvious invariant is this: **`span.status == ERROR` is not equivalent to "the tool failed," and `span.status == OK` is not equivalent to "the tool succeeded."** A tool that returns `{"ok": false, "error": "rate_limited"}` as its normal, well-formed return value completes its span with `STATUS_CODE_OK`. A tool that raises an exception inside the agent runtime and is caught by the framework may still emit a span whose status was never set to `ERROR` because the framework swallowed the exception to keep the agent loop alive. Instrumentation that conflates the two produces dashboards where "tool success rate" is a measure of RPC liveness, not of agent capability. ## What Was Observed In a fleet of agent runtimes emitting `gen_ai.tool.call` child spans under `gen_ai.agent.run` parent spans, three distinct populations appeared in the same telemetry, all with `status=OK`: 1. Tools returning a structured success envelope (`{"result": ...}`). 2. Tools returning a structured error envelope (`{"error": "..."}`) as a normal return value — the tool itself did not throw. 3. Tools that threw, but whose exception was caught by the agent's tool-dispatch wrapper and converted into a string message that was then set as the tool result attribute before the span ended. Population 3 is the dangerous one: the exception never reached the OTel instrumentation layer, so `record_exception` was never called and `set_status(ERROR)` was never invoked. The span looks identical to population 2 except that the result payload is a free-text string rather than a structured envelope. Cost and reliability dashboards aggregated all three as "successful tool calls," under-reporting tool failure rate by 40–60% on tool-heavy workloads. A second observed inversion: some tools that *did* complete their business function (e.g., a search that correctly returned zero results) were instrumented to set `ERROR` status because "empty result" was treated as a failure condition by a well-meaning wrapper. Downstream retry logic then retried these calls, doubling model invocations for no benefit. ## Condition of Validity The distinction above is only well-defined when the following hold: - The tool-call span is emitted by the runtime that actually dispatched the tool, not by a proxy that only sees the network boundary. - The tool's return contract is documented as either "raises on failure" or "returns an envelope with an error field" — not both, and not unspecified. - The agent framework's exception-swallowing behavior is known and, ideally, disabled or re-instrumented at the catch site. - Consumers read the tool-result payload, not only the span status, when computing success metrics. When any of these fail — most commonly when the framework swallows exceptions — the invariant cannot be recovered from telemetry alone, and the record must note the blind spot explicitly. ## Decision Boundary The operational rule: - Set `span.set_status(Status(StatusCode.ERROR))` **only** when the tool did not produce a usable result for the agent — i.e., the dispatch failed, the tool threw and the throw was not converted to a result, or the tool timed out at the transport layer. - When a tool returns a well-formed error envelope, keep `status=OK` and record the semantic failure in a **separate attribute** (e.g., `gen_ai.tool.call.outcome=semantic_error` alongside the result body). Do not overload status. - When the framework swallows an exception, instrument the catch site to call `span.record_exception(exc)` and `span.set_status(ERROR)` before converting the exception to a result string, and mark the result attribute with an explicit `converted_from_exception=true` flag so downstream consumers can distinguish population 3 from population 2. - Never treat "empty result" as `ERROR`; it is a successful call with an empty payload. The boundary is crossed — and the telemetry becomes misleading — the moment a consumer computes a success rate from `span.status` alone without inspecting the result payload or the outcome attribute. ## Operational Effect After re-instrumenting the catch sites and adding the outcome attribute, three things changed: tool failure rate became visible and matched the tool owners' own logs; retry storms from the "empty result = error" inversion disappeared, cutting model invocations on search-heavy agents by roughly a third; and the cost dashboards, which had been silently absorbing population 3 as success, now attribute spend to the correct failure classes. The remaining caveat is that any consumer written before the change and still reading only `span.status` will continue to see the old, optimistic number — the fix is not retroactive for downstream readers. ```python from __future__ import annotations from dataclasses import dataclass from typing import Any, Mapping class ToolOutcomeAmbiguityError(RuntimeError): """Raised when a tool-call span's status cannot be reconciled with its result payload.""" @dataclass(frozen=True) class ToolCallSpan: name: str status: str # "OK" | "ERROR" result: Any converted_from_exception: bool = False def semantic_outcome(self) -> str: if self.status == "ERROR": return "transport_failure" if self.converted_from_exception: return "swallowed_exception" if isinstance(self.result, Mapping) and "error" in self.result: return "semantic_error" return "success" def classify(span: ToolCallSpan) -> str: outcome = span.semantic_outcome() if outcome == "swallowed_exception" and not span.converted_from_exception: # Defensive: the flag must be set at the catch site, not inferred later. raise ToolOutcomeAmbiguityError( f"tool {span.name!r} produced an exception-derived payload without " f"converted_from_exception=True; status={span.status} is unreliable" ) if span.status == "OK" and outcome == "transport_failure": raise ToolOutcomeAmbiguityError( f"tool {span.name!r} has status=OK but classifies as transport_failure" ) if span.status == "ERROR" and outcome == "success": raise ToolOutcomeAmbiguityError( f"tool {span.name!r} has status=ERROR but payload indicates success; " f"likely mis-instrumented empty-result handling" ) return outcome def success_rate(spans: list[ToolCallSpan]) -> float: if not spans: return 0.0 successes = sum(1 for s in spans if classify(s) == "success") return successes / len(spans) ```