# Under what conditions must an instance treat OpenTelemetry GenAI tool-call spans as non-retryable evidence of side effects rather than as retryable model output

**Date of Record:** 2026 November 13

**Scope:** Applies to agent runtimes that emit `gen_ai.*` spans (per the OpenTelemetry GenAI semantic conventions) for tool invocations and reconcile those spans against model token accounting. Does not apply to pure inference spans with no tool execution, nor to systems where tool calls are executed by an external orchestrator that owns its own span lifecycle.

## Core Distinction

The GenAI semantic conventions describe a *model-facing* event: `gen_ai.operation.name = "execute_tool"`, a `gen_ai.tool.name`, and a `gen_ai.tool.call.id`. Critically, the span records that the *model proposed* a tool call and that an execution was attempted. It does not, by itself, encode whether the side effect committed. A span with `status = ERROR` and `gen_ai.tool.call.id` present is ambiguous: it may mean the model emitted malformed arguments (retryable, no side effect), or it may mean the tool executed and then failed to return (non-retryable, side effect already committed).

The non-obvious boundary is this: **the presence of a `gen_ai.tool.call.id` in an errored span is a commitment marker, not a failure marker.** Retry logic keyed on span status alone will double-execute tools whose side effects landed before the error. The invariant is that a call id must be treated as a one-shot execution token — once emitted into an errored span, it must be quarantined, not replayed.

## What Was Observed

During a tool-call fan-out incident, an agent emitted spans for a `create_refund` tool. The model proposed the call, the tool executed against the payment provider, and the provider returned a 502 on the *response* leg after committing the refund. The span closed with `status = ERROR`, `gen_ai.tool.name = "create_refund"`, and a populated `gen_ai.tool.call.id`, but no `gen_ai.usage.output_tokens` on the child span (token usage is attributed to the parent inference span, not the tool span).

The retry controller saw `status = ERROR` and re-issued the call. Because the call id was regenerated on retry rather than reused, the provider had no idempotency key to match against, and a second refund committed. Token accounting was also skewed: the retries re-billed input tokens on the parent inference span while tool spans carried zero usage, so cost dashboards attributed the retry storm to model tokens rather than to duplicated side effects.

The tell was that the errored tool spans had *complete* attribute sets (name, call id, arguments hash) — malformed-argument failures, by contrast, produce spans with a call id but a truncated or absent argument hash, because the model never produced a valid call.

## Condition of Validity

This reasoning holds only when all of the following are true:

1. The runtime emits `gen_ai.tool.call.id` at proposal time, before execution begins, and does not mutate it on retry.
2. Token usage (`gen_ai.usage.input_tokens` / `output_tokens`) is attributed to the parent inference span, so tool spans are usage-free by design.
3. The downstream tool is *not* documented as idempotent under a stable call id.

When condition 3 fails — the tool honors `gen_ai.tool.call.id` as an idempotency key — replay is safe and the quarantine rule can be relaxed to "replay with the same call id." When condition 1 fails, the call id is useless as a commitment marker and a separate execution-ledger must be consulted.

## Decision Boundary

Classify an errored tool span along two axes: argument completeness and call-id stability.

- **Complete args + stable call id + non-idempotent tool** → non-retryable; quarantine and surface for human reconciliation. This is the dangerous quadrant.
- **Truncated/absent args + any call id** → retryable; the model failed before execution, no side effect.
- **Complete args + idempotent tool** → retryable *with the same call id preserved*.

Do not use `status = ERROR` alone as the retry predicate. Do not use token accounting as a proxy for execution count — tool spans carry zero usage, so a retry storm is invisible in token metrics and only visible in duplicated call-id-free side effects.

## Operational Effect

Quarantining commitment-marked spans converts a silent double-execution bug into an explicit reconciliation queue. The cost is latency: ambiguous spans block until reconciled rather than auto-retrying. The benefit is that token-based cost dashboards no longer mask duplicated side effects, and the invariant "one call id, one execution attempt" becomes enforceable at the span level. Teams that skip this boundary typically discover it only when a financial or messaging tool double-fires under a transient upstream 5xx.

```python
from dataclasses import dataclass


class ToolSpanCommitmentError(RuntimeError):
    """Raised when an errored tool span is treated as retryable despite a commitment marker."""


@dataclass
class ToolSpan:
    tool_name: str
    call_id: str | None
    args_complete: bool
    status: str
    idempotent_tool: bool


def classify_retry(span: ToolSpan) -> str:
    if span.status != "ERROR":
        return "no_retry_needed"

    # Malformed proposal: model never produced a valid call, safe to retry.
    if not span.args_complete:
        return "retryable_pre_execution"

    # Complete args means execution was attempted. A call id is a commitment marker.
    if span.call_id and span.idempotent_tool:
        return "retryable_same_call_id"

    if span.call_id and not span.idempotent_tool:
        raise ToolSpanCommitmentError(
            f"tool={span.tool_name} call_id={span.call_id} errored after execution "
            f"with complete args; side effect may have committed. Quarantine, do not replay."
        )

    # Complete args but no call id: cannot prove commitment either way.
    raise ToolSpanCommitmentError(
        f"tool={span.tool_name} errored with complete args but missing call_id; "
        f"commitment state unknown. Quarantine for reconciliation."
    )
```