# Why does an instance of a runaway agent loop appear healthy under GenAI semantic-convention trace-level anomaly detection **Date of Record:** 2026 November 17 **Scope:** Applies to trace-level anomaly detection built on OpenTelemetry GenAI semantic conventions (`gen_ai.*` attributes, `execute_tool` spans, `chat` spans) for autonomous agents that invoke tools in a loop. Does not apply to single-shot RAG pipelines, batch inference, or systems where tool invocations are not instrumented as child spans of a root agent span. ## Core Distinction A runaway agent loop is a *structural* pathology — the same state transitions repeat with a monotonic budget drain — not a *distributional* one. Trace-level anomaly detectors typically operate on per-span or per-trace aggregates: latency, token counts, span counts, error rates, and attribute-value distributions. Those aggregates are computed over a trace *after* it closes, or over a rolling window of completed spans. The failure mode is that a loop of length N produces per-trace statistics that lie inside the normal envelope, because the loop's defining property — *repetition of an identical structural motif* — is invisible to scalar aggregates that discard ordering and identity. The non-obvious invariant: **an agent loop that repeats a fixed tool-call motif with stable arguments will produce a trace whose latency p50, token-count distribution, and error rate are all within one standard deviation of a healthy trace of the same length.** The detector is not wrong about the numbers; the numbers do not encode the pathology. ## What Was Observed A `gen_ai.agent` root span with `gen_ai.operation.name=invoke_agent` spawned a repeating motif: `chat` span → `execute_tool` span (`gen_ai.tool.name=search_docs`) → `chat` span, for 340 iterations before the wall-clock budget killed it. Observed properties of the completed trace: - **Span count** was high but not anomalous: 681 child spans, within the p99 of legitimate multi-step research traces (which legitimately reach 600+ spans). - **Per-span latency** was nominal. The tool was a fast in-process index lookup (~40 ms), so `execute_tool` p95 latency was *lower* than the fleet median, not higher. A latency-anomaly rule flagged nothing. - **Token counts per `chat` span** were small and tightly clustered, because the model kept re-emitting a near-identical short planning message. The token distribution was *less* variable than healthy traffic, so variance-based detectors saw it as clean. - **Error rate was zero.** No exceptions, no `status.code=ERROR`, no retries. The loop was succeeding at every step. - **Attribute cardinality** was the tell, but only if you computed it: `gen_ai.tool.call.id` had 340 distinct values (correctly unique per call), but the tuple `(gen_ai.tool.name, hash of tool arguments)` had cardinality 1. Every one of the 340 tool calls passed byte-identical arguments. The trace was healthy on every scalar axis and pathological on exactly one relational axis: repeated identical `(tool_name, arguments)` tuples within a single trace. ## Condition of Validity The detection gap holds when all of the following are true: 1. The loop's repeated step is *deterministic in its arguments*. If the agent perturbs its query each iteration (e.g., appending "again" or varying a page offset), argument-hash cardinality rises and the signature degrades. Loops that mutate arguments require n-gram structural detection instead. 2. The per-iteration cost is *below* the anomaly threshold for individual spans. A loop of expensive calls trips latency or token alarms on the first few iterations; a loop of cheap calls does not. 3. The trace is evaluated by *aggregate* rules (counts, means, variances, error rates) rather than by *sequence* rules (repeated-subtree detection, motif cardinality, state-transition entropy). 4. The budget that eventually stops the loop is enforced *outside* the traced operation (wall-clock, external supervisor) and therefore does not itself appear as an error span. When any of these fail — particularly (1) or (4) — the loop becomes visible through a different channel, and this record's specific trap does not apply. ## Decision Boundary The boundary between "healthy long trace" and "runaway loop" is not a threshold on any scalar. It is a predicate on the *multiset of structural motifs* within the trace: - **Healthy:** structural motif cardinality grows roughly linearly with span count; repeated identical `(tool_name, canonicalized_arguments)` tuples occur at most a small constant number of times (legitimate retries, pagination with distinct cursors). - **Runaway:** a single `(tool_name, canonicalized_arguments)` tuple recurs more than K times (empirically K≈5 for research agents, K≈2 for transactional agents where idempotent re-invocation is never legitimate), *independent of* total span count, latency, or error rate. The decision must be made on canonicalized arguments (sorted keys, normalized whitespace, stripped volatile fields like timestamps and request IDs), because raw argument hashes will not collide under trivial serialization jitter. The detector must run *within* the trace's own span tree, not across traces, because each iteration is a child of the same root. ## Operational Effect Deploying scalar-only trace anomaly detection over GenAI semantic conventions yields a detector with a *blind spot precisely at the failure mode that consumes the most budget*: cheap, successful, repeated tool calls. Operationally this means: - Runaway loops are caught by the external budget kill, not by observability — so mean time to detection equals the budget window, and cost per incident equals the full budget. - Post-hoc analysis must be run to explain incidents that the detector "should" have caught, eroding trust in the anomaly pipeline. - The fix is to add a trace-internal motif-cardinality rule keyed on canonicalized tool arguments, evaluated at trace close, with the K threshold set per tool class. The following guard encodes the boundary as an executable check that can be wired into a trace post-processor. ```python import hashlib import json from collections import Counter from typing import Any, Iterable class RunawayLoopDetected(RuntimeError): """Raised when a single trace repeats an identical tool-call motif beyond its permitted cardinality.""" pass def _canonical_tool_arguments(arguments: dict[str, Any]) -> str: # Sort keys, drop volatile fields that never indicate semantic change. volatile = {"request_id", "trace_id", "timestamp", "nonce", "attempt"} cleaned = {k: v for k, v in arguments.items() if k not in volatile} return json.dumps(cleaned, sort_keys=True, separators=(",", ":")) def _motif_key(span: dict[str, Any]) -> str | None: if span.get("gen_ai.operation.name") != "execute_tool": return None tool = span.get("gen_ai.tool.name") raw_args = span.get("gen_ai.tool.call.arguments", {}) if tool is None: return None digest = hashlib.sha256(_canonical_tool_arguments(raw_args).encode()).hexdigest()[:16] return f"{tool}:{digest}" def assert_no_runaway_loop( spans: Iterable[dict[str, Any]], max_repeats: int = 5, ) -> None: """ Enforce the trace-internal motif-cardinality invariant. Scalar aggregates (latency, tokens, error rate) cannot see this pathology; only repeated identical (tool_name, canonicalized_arguments) tuples reveal it. """ counts: Counter[str] = Counter() for span in spans: key = _motif_key(span) if key is not None: counts[key] += 1 for motif, n in counts.items(): if n > max_repeats: raise RunawayLoopDetected( f"Runaway agent loop: motif {motif!r} repeated {n} times " f"in a single trace (limit {max_repeats}). " f"Scalar trace metrics are expected to be nominal; " f"this predicate is the only signal." ) ```